All posts by rangler2

OneTrust / CookiePro script replacement type=module

Uncategorized

OneTrust and CookiePro are two versions of the same cookie consent product from OneTrust. We use one of these tools on many websites to allow users to consent to cookies, there is a control panel where you can configure the UI etc and it is fairly unobtrusive compared to some cookie consent products.

Typically once you have configured and added the cookie consent banner to your site, it won’t actually stop cookies (especially 3rd party ones) unless you configure your site not to load 3rd party scripts until the requisite cookie category consent has ben provided by the user.

There is a load of documentation on ways to do this.

The simplest, for 3rd party script tags, being the JavaScript Type Re—Writing. Instead of

<script>

or

<script type="text/javascript">

you change this to, for example,

<script type="text/plain" class="optanon-category-C0002">

and OneTrust will automatically load the script after the user accepts cookies for the category specified (C0002 = performance category) both on 1st and subsequent page loads.

However, what if your 3rd party script was a module i.e.

<script type="module">

I could not find any documented means to get OneTrust to load the script with the correct type=module, so had to improvise with this script instead:

 <script type="text/plain" class="optanon-category-3">
    const myscript = document.createElement('script');
    myscript.src = 'https://3rdparty.domain.com/my-app.js';
    myscript.type = 'module';
    document.body.appendChild(myscript);
</script>

This seems to work perfectly well and better than other hacky workarounds I tried such as loading the script in the OptanonWrapper function or using the OneTrust.InsertScript or OneTrust.InsertHTML helper methods.

Azure Private DNS will break your network if used incorrectly

Azure

I am not a networking expert but in configuring some Azure cloud services I came across the need to use Azure Private DNS to create a private DNS zone for an App Service Environment v3 (Isolated web app that sits within a vnet).

This is something that is needed to be able to connect to the web apps that are hosted in the ASE from elsewhere in the virtual network (e.g. other web apps in the ASE), without needing hosts file entries (which are impossible to create on PaaS services).

Then I realised that my web app also needs to connect to other existing web apps which are hosted internally. These apps have been setup on Windows VMs using IIS with the proper HTTPS binding with their proper URL (public traffic is routed via an Application Gateway). In this particular restricted environment, outgoing internet HTTP/HTTPS requests are restricted to pre-approved domains. Hence on existing VMs we have some hosts entries so that they can access each other via their internal IPs.

My plan was: Create an Azure Private DNS zone and then create A records matching the app URLs with the internal IP of each app.

However, it turns out that once you create a Private DNS Zone, all public records beneath this are no longer accessible from resources within your vnet. You would have to duplicate all the public DNS records to be able to have a private DNS zone for a top level domain.

I did read about Split Horizon / Split Brain DNS and I was hoping that if a DNS entry isn’t resolved by the Private Zone then it gets recursively resolved by Public DNS but it isn’t the case. I believe you could to use Azure DNS Private Resolver to do this, which is much cleverer, but a much bigger thing to add to a vnet.

Here’s an example.

From a VM in the vnet in question I can do some DNS queries for my company’s public domain:

Then, in the Azure Portal (in an account / subscription that has nothing to do with Great State, but has a virtual network and VMs already) I can create a Private DNS zone for greatstate.co – just like the documentation about Split-Horizon functionality suggests doing for contoso.com :

I have added a test.greatstate.co DNS record so I can be sure it is working.

I then link this up to my vnet from the Virtual network links tab.

Then go back to my VM, flush the DNS and I can query the test.greatstate.co DNS record. Unfortunately, I can no longer query greatstate.co or www.greatstate.co so I have totally broken DNS for the entire domain within the entire virtual network!

This is such a dangerous thing that I’m surprised there isn’t a warning in the docs or in the Azure Portal in the creation of a Private DNS zone.

This page does suggest that it can have consequences since Microsoft have blocked their own domains being used as Private DNS zones!

How to check and rebuild Examine indexes on each node of an Umbraco Azure Autoscaled web app

Umbraco

If you run your Umbraco 7+ on Azure, and you have split out CM and CD so that /umbraco is not accessible on the CD site (i.e. www) then you may have wondered how to rebuild the Examine indexes if you ever had to. Or even how to check the status.

Of course you could enable Umbraco access on CD but this is not ideal, as it exposes the Umbraco admin interface to the world. Even if you do this temporarily and then change it back there is a risk that you forget and/or break the website in doing so. You could enable it just from your IP address.

Even if you do this, when you go to the Developer > Examine Indexes tab, if your app is scaled up 2+ instances then each time you refresh it will go to a different server and be difficult to tell which server(s) indexes you are looking at / rebuilding.

To help with this I made a quick script that can be dropped into an Umbraco 7 site.

The only thing you also have to do is to add its URL into the umbracoReservedUrls setting in web.config.

It is very basic and ugly, but it shows you the important information – – which server you are looking at, and the state of the indexes (how many documents).

You can compare this against the expected state of each index from the CMS (where you can reliably rebuilt the indexes through the backoffice first to ensure they are complete).

If any index is incomplete on one or more nodes, the tool lets you rebuild it by specifying the machine and index name and then press Rebuild. In case the next request goes to one of the other nodes, there is a warning and you can click Rebuild again until your request goes to the correct node. Then just refresh a few times until you see that the index count on the given node is what you expect.

This simple tool allowed me to rebuild a corrupt index and more importantly be confident that the index state on all the active nodes is the same and complete.

The code can be found here, just save the .aspx file into the website (add it to umbracoReservedUrls if needed).

Have only tested this on v7.6.4 (and I know, I need to upgrade, which will probably solve the indexing problems!) it should at least work on newer v7 and possibly newer versions.

Reboot Node if Needed in Azure Automation State configuration (DSC)

Azure

In Azure Automation when onboarding a server you have a choice whether to “Reboot Node if Needed” which many tutorials will suggest you should tick.

However you might wonder what about production servers, do you really want them to reboot themselves?

Of course you could initially onboard a server with the Reboot ticked if needed for initial setup and then re-onboard it with it unticked so that once in production it doesn’t get rebooted. But what if you forgot to change it? For this reason I’d rather onboard Prod servers without automatic reboot.

So I wondered, what actually is the experience if you don’t tick the box but the node requires reboot? Is it still possible to apply configurations that require reboot but manually rebooting? How would you know when the reboot is required?

What I found was the node gets “stuck” in “In progress” state:

If I click into the node then (after some time has passed) I see that no status is shown even though reports are coming through (normally these would show Compliant or Failed):

If I click on either of these then it says:

So there are no failures logged it just is unable to state if the configuration is fully applied or not.

On the server I tried to interrogate the DSC status:

It isn’t totally clear that the node requires rebooting. But if you are aware that this is the reason the configuration “has not converged yet” then it is easy enough to manually reboot the node.

After reboot, the node goes to (in my case) Failed or Compliant like usual.

So it is fine to not allow reboot keep in mind that the node will show In Progress indefinitely until rebooted if this is required by the configuration. At least this is the case with my example that is waiting for WebAdministration role to be installed

Poor man’s approach for diffing your Sitecore 9 database

Sitecore

During an upgrade we needed to compare the core database with the out of the box one to confirm what changes had been made, in case they hadn’t been checked in with Unicorn.

There is a great tool called Razl that I recommend you look at. It used to provide a free trial but not any longer.

As we only wanted to do a quick diff I tried an alternative approach, using the built-in serialisation tools.

  1. Set up a Sitecore instance using the target database
  2. Log into Sitecore as admin
  3. Right click the content editor ribbon and enable Developer
  4. Choose the portion of the tree that you want to diff, click Serialize Tree
  5. Download the contents of the App_Data/Serialization folder (or just the sub folder that represents the part of the tree you are interested in) it might help to clear out the Serialization folder if you aren’t using it and want to just compare it wholesale
  6. Run through steps 1 – 5 using a Sitecore instance pointed at the source version of the database you want to diff (for example, the vanilla Sitecore database for your Sitecore version)
  7. Use a tool such as WinMerge to compare the two folders, and hide the identical items. It is actually quite easy to navigate and diff to see the changes that have been made.
  8. Diff any files that represent items to see what has been changed. If still unclear, use Sitecore to load up the item in question and see the changes in the content editor.

Happy diffing!

Speeding up Sitecore 9.1.1 Experience Editor with Limited Page Editor role

Sitecore

After an upgrade from Sitecore 9.0 to 9.1.1 we found that the Experience Editor load time had gone from 8 seconds to over a minute! The time is all spend loading Ribbon.aspx.

Sitecore have been unable so far to determine the cause of the slowdown, but various posts imply that other people have the same issue either on new or upgraded 9.1 sites, and that a good way to speed this up is to tweak the page editor ribbon options.

I spend some time looking at the built-in roles that might help.

I found that any user that inherits from sitecore\Author is extremely slow (~70 seconds). The tabs include “Optimization” which is not even needed by our site which has xDB Disabled!

It looks like this is due to sitecore\Author inheriting from Analytics roles:

  • sitecore\Author
    • sitecore\Analytics Testing
    • sitecore\Analytics Personalization
    • sitecore\Sitecore Client Authoring

If I use sitecore\Sitecore Client Authoring instead then the load time goes down to 40 seconds and it no longer includes the Optimization tab.

This implies that the Optimization tab was adding about 30 seconds. But 40 seconds is still too slow.

I found that by adding the role sitecore\Sitecore Limited Page Editor in addition to either of the above roles then it loads in 15 seconds but only the ribbon contains 2 tabs – Home and Versions. However the tick boxes on the View tab are present on the Home tab.

The only issue here is that the Add Component button is greyed out. I found that this is due to it being Denied read access in the item in the core database – /sitecore/system/Settings/Security/Policies/Page Editor/Can Design

Once I removed this, the design features are available and the experience editor appears perfectly usable, and much faster.

If / when we find a better way to resolve this without limiting the features, I will update this post!

Multiple Workstreams Git Branching Strategy and VSTS Build/Release

Azure DevOps, Git

After moving from Subversion to Git we struggled to adapt our old branching strategy, which was to do all work in trunk and then cherry pick bunches of commits into a release branch for QA/deployment. Git prefers to merge whole branches at a time, while cherry pick is supported it results in a different commit being made in the target branch and gets messy very quickly.

For most projects we just moved to a more sprint-based approach using gitflow or gitlab flow, did work in feature branches, pull request into master when dev complete, then master is deployed to test site, when a sprint worth of work is complete, either deploy straight from master or a release branch.

However, a few projects didn’t suit this because we always have a number of workstreams on the go at the same time: a few may need to be deployed to test together, but then end up going live one at a time in another order. Feature flags and automated tests are great but sometimes bad code just gets into master, and having this go live would be a nightmare. I concluded:

  • Work must remain in feature branch until it is GO LIVE READY:
    • Code reviewed by another dev
    • Automated tests pass
    • Manual QA pass
    • Approved by product owner

So we can set branch policy on master to require a few people to approve, and a working pull request build which runs unit tests. With that in mind we now have master is basically live, and feature branches in various states. How to test them?

  • Deploy one branch at a time to test site. Impractical if multiple workstreams need to be visible at once.
  • Set up an environment for each branch ready to test. A lot of effort and potentially cost.
  • Merge the “ready” branches into a copy of master then deploy this to the test site.

Option 3 was the only option for some projects. But the thought of maintaining a ‘qa’ branch by manually merging the correct features into it sounds terrible. So we automated it.

In VSTS we have a Build that is manually triggered, it checks out the latest master branch then runs this powershell:

git fetch origin
git reset
git config --global user.name GitTask
git config --global user.email [email protected]
git checkout -b uat/$(Build.BuildNumber)
git merge --no-ff $(branches)
if ($LastExitCode -ne 0) {
  throw "Merge Failed"
}
git push --set-upstream origin uat/$(Build.BuildNumber)

There is a build variable which lists the branches ready to be tested e.g. “origin/feature/123 origin/feature/245 origin/bugfix/123”. This is manually edited when a new branch needs to be included for test, then a build queued. This is doing an ‘octopus merge’ i.e. merging multiple branches into one.

It checks this into a branch named based on the build number e.g. uat/{date}-{count} or whatever you like. This is so you can manually interrogate the result of the merge (and what has been deployed) and also to split the merge from the actual build. To enable the script to “git push” I had to edit the repository security and enable ‘Create branch’ and ‘Contribute’ permission for the ‘Project Collection Build Service’.

Then there is a normal Build which compiles the code and generates an artefact (web deploy package in our case) which is set to trigger on uat/* as well as master (the same build generates the artefact from master branch which is deployed to live).

Then we have a Release which picks up this build (also filtered on branch uat/*) and deploys it to the QA site.

So if a developer updates any of the branches in QA they only have to queue the automerge build.

Once a particular change is approved and the pull request closed, this triggers the build from master, which triggers the start of a deployment pipeline to deploy to staging and then live (with manual approval steps).

This has worked pretty well so far, and solved the problem of multiple workstreams for us. I would only suggest something like this if you have tried to work in a normal git branching strategy and it isn’t working out. As it could come back and bit us if we let the feature branches get too large, the merge might continually fail, or we might get lazy at closing pull requests, etc. But I thought I’d share in case this helps anyone with a similar issue.

Add Page Views column to the table on Sitecore Analytics Reports

Sitecore

On many pages of the Sitecore Analytics dashboard, there is a table of visit data.

This by default is sorted by page views descending, but there is no page views column! The Visits column is actually the number of visits where at least one page view hit the page in question.

We were asked to add Page Views to this table, and it turned out to be incredibly easy.

In the Core database, go to /Sitecore/client/Applications/ExperienceAnalytics/Common/System/ListControl and you’ll see all the headings from the above table are items under this one:

 

Simply duplicate the Visits item and change the header to “Page views” and data field to “pageViews” as per the above screenshot.

Hey presto:

Also worth pointing out you can set the default sort option here under the relevant ExperienceAnalyticsListControl Parameters item (see my previous post for more detail on customising the Analytics pages):

 

 

Deploying Sitecore Unicorn Items to Azure Web App using VSTS Build and Release

Azure, Azure DevOps, Sitecore

For years we have been using Unicorn to manage and deploy our developer-owned Sitecore items using TeamCity. Nowadays, we have moved to using VSTS for source code, build and release management, and Microsoft Azure for web app hosting. We struggled to set up a process akin to our TeamCity deployment process and Darren Guy’s epic blog series only covers Octopus Deploy.

I won’t go into setting up Unicorn as this is covered in the documentation. I will assume you have got it saving serialized files into source control and just outline the solution we have used to deploy the items.

We decided to put the Unicorn files into /App_Data/Unicorn so it is beneath the Webroot so can be deployed to, but protected from the public.

1. Configure the build to include Unicorn files

We used a Copy Files step to copy the Unicorn folder from source control into the artefact staging directory. This means the files are available at Release time. We’re copying it into a new folder “UnicornWWW” in the location under the website path that we want to deploy the items to (App_Data/Unicorn).

We also zip this folder up into a Unicorn.zip which contains the path within the website inside the zip, and delete the temporary folder as we don’t need it anymore.

Now the UnicornWWW.zip is a build artefact available at release time. You can check by going to the completed build summary page and checking the Artefacts tab, where the file should be listed.

2. Deploy the Unicorn files

This was tricky as we want to not only deploy the new files but also delete any old ones that have been removed from source control. We already had an Azure App Service Deploy step to deploy our web deploy package to the web app. What we did was add a post deployment script to this step which removes the Unicorn folder:


Then we have a follow up deployment which deploys the UnicornWWW.zip directly to the web app. As it has the App_Data/Unicorn folder inside it, the Unicorn files are deployed to the right place and the previous version have been deleted beforehand.

I believe it should be possible to do a manual msdeploy call inside a Powershell script where you could target a subfolder and do ‘sync’ to delete existing files, much like we do with TeamCity, but we haven’t got that working quite yet as not sure how the publish credentials would be taken out of the service principal setup that connects VSTS and Azure – it “just works” when you use the Azure App Service Deploy task so for now this is what we are sticking with.

3. Sync the Unicorn files

First we need a copy of the Unicorn Powershell API scripts available to the release agent. To do this we go back to the build and add a step to copy the Unicorn Tools PSAPI folder from the nuget package location into the build artefact folder:


We then use a Powershell step which runs on the Release agent, which triggers Unicorn sync using the Unicorn Powershell API. This is based on the sample.ps1 from the PSAPI folder. Here we specify the environment URL + /unicorn.aspx and the shared secret that is configured as per Unicorn.UI.config.


The URL and secret can be made environment variables and then you have a fully automated deployment of Unicorn items to your Azure web app environment!

Customising Sitecore Experience Analytics Dashboard

Sitecore

We had a request to customise the default Experience Analytics dashboard on a Sitecore 8.1 site. With a little guidance from Sitecore we have been able to do this easily without development, but it is a bit of a convoluted process so I thought I would outline it here.

If possible, get your site live with Analytics configured correctly and capturing data well before you start looking at customising it, as you can then browse around the default analytics pages and see the real data that is captured. You can then give an analyst access and ask them what changes would be useful.

The default dashboard looks a little like this, on a site that hasn’t had any Analytics customisation done:

All the blocks apart from are ‘by value’ which means there is no data because no profiles / personas / experience scoring has been configured. Therefore, our analyst asked if the dashboard could be changed to include more common analytics like you might find on a Google Analytics dashboard.

  • Total Visits (line chart)
  • Top Referring Site (bar graph)
  • Top Entry Pages (bar graph)
  • Pageviews + Pageview/Visit (line chart)
  • Top pages (bar chart)
  • Top pages over time (line chart)
  • Top 10 pages (table)

These are all graphs that she found in other Analytics report pages, so all we had to do is figure out how to remove the existing charts from the dashboard and copy these from the other pages, creating a new dashboard layout.

The most relevant Sitecore documentation discusses how to create a new report page with a single chart. Using this and a bit of Sitecore knowledge you can figure out how to edit an existing complex dashboard.

The analytics reports structure is stored as Sitecore items in the core database. You can view it by using Sitecore Desktop (link from the Sitecore launchpad) and switch to Core DB using the link at the bottom right:

Then open Content Editor and browse to /sitecore/client/Applications/ExperienceAnalytics/Dashboard

You can see that beneath Dashboard are all the report folders (Audience, Behaviour, etc – which are just common Folders) and report pages (Overview, Devices, etc – using template /sitecore/client/Speak/Templates/Pages/Speak-DashboardPage)

Each report page has a PageSettings folder (template = /sitecore/client/Speak/Templates/Pages/PageSettings) that contains an item underneath it for every chart on the page. These items allow you to set the chart title and any other settings such as which metrics / dimensions to use on the chart.

Review the Layout Details for any page with multiple charts and you can see it is made up of a series of Rows and Columns which define the page layout, and renderings for each chart.

Click Edit and you can see the placeholder IDs used – now you can see how the page is put together. RowPanels are rendered one above the other (in Main.Content placeholder) and each RowPanel automatically defines a placeholder “RowPanel X.Content” where X is the row number. Then ColumnPanels are inserted into the RowPanel placeholders to set up however many columns you want to break that row into. In this example, one full width row, then two 50/50 rows.

The chart renderings themselves are inserted into the ColumnPanels using autogenerated placeholders based on the row and column number e.g. “ColumnPanel 3.2.Content” means this rendering will be shown in row 3 column 2.

To edit this becomes more interesting because you can’t simply edit this in Sitecore – Edit gives you a server error, and Change gives you an empty list of renderings:

So based on the Sitecore article we have to switch to Sitecore Rocks which is able to edit these. Install and connect to the Sitecore instance, then expand Core database to the same location. Select a report page, Right-click, Tasks, Design Layout:

Now you can edit the configuration of rows/columns by moving/adding/removing RowPanel and ColumnPanels, editing the Placeholder of each. Note that each ColumnPanel has a ‘GridColumns’ field set to 12 for full width, 6 for half width, 4 for 1/3 width, etc. Each RowPanel seems to be used for a single Row, not floating multiple charts into it.

The charts themselves use one of a number of Renderings all starting with ‘ExperienceAnalytics’ – you can copy/paste these using right click options from one page’s layout to another, or Add Rendering to pick one to add from scratch:

Each chart needs to have its Datasource set to a PageSettings child item using the matching parameters template:

Easiest is to find and copy the existing settings item from another report page, under the dashboard page, and then edit the datasource of the chart rendering item to use the new settings item.

Save regularly and reload the dashboard page, to see your changes. Here is my modified dashboard page where I have different charts: