Category Archives: Sitecore

Investigating Sitecore 10.2 XP on Azure PaaS Updating screen

After a normal, successful, code deployment to a Sitecore 10.2 XP site hosted in Sitecore Managed Cloud (PaaS – App Services) we found this screen on the CM instance “Stay tuned we are updating…”

TL;DR how to fix:

  1. Browse to SCM for the CM site e.g. mc-012345-cm.scm.azurewebsites.net
  2. Go to Debug console
  3. Delete site\wwwroot\app_data\Maintenance.lock
  4. Go into site\wwwroot
  5. Run this via the console: `app_data\tools\installjob\Sitecore.Cloud.Integration.Bootload.InstallJob.exe .`

After searching for this I found some helpful posts including https://www.asmagin.com/posts/2019/01/stuck-on-stay-tuned-page-after-scwdp-package-deployment/ but as this is so peculiar I wanted to keep notes of my own investigation into the reason for this screen, which seems to be caused by an AppOffline.htm file in my site folder… but why is it there?

First I went to scm and attempted to manually run the bootloader as per the above blog, which sure enough exists in the app_data\tools\installjob folder (must be part of Sitecore Managed Cloud 10.2 deploy as we haven’t put it there).

Looks like there is a file App_Data\Maintenance.lock , so I deleted that and then retry:

Well it seemed to work this time and the site came back up. So that is a bit of a mystery.

Later on I found “InstallJob.log” files in /home/LogFiles (thanks to pointers in \App_Data\tools\InstallJob\Sitecore.Cloud.Integration.Bootload.InstallJob.exe.config)

Sure enough seems like there was an aborted installation job yesterday that installed the AppOffline file even though according to the log it was supposed to be “exiting installer”

Then the log from this morning shows it failed to run a few times but did nothing until 9:12 which was when I deleted the Maintenance.lock file and re-ran it manually.

Digging deeper into what this all is:

In our case we have Sitecore Headless Services 20.0.2 installed and to ensure the correct files are deployed to each environment I am deploying the files from the “Sitecore Headless Services Server XP 20.0.2 rev. 00545.scwdp.zip” file , which turns out to include the JSSSCCLP.sccpl file inside App_Data\Transforms. This is a zip file as described in https://doc.sitecore.com/xp/en/developers/sat/28/sitecore-azure-toolkit/the-structure-of-an-sccpl-transformation.html

This particular zip contains one web.config XDT transform which is to add the jss media handler:

The docs mention bootloader in relation to ARM templates (which we aren’t using since the deployment was done by Sitecore Managed Cloud) but don’t mention any detail about how it processes files on app startup https://doc.sitecore.com/xp/en/developers/sat/28/sitecore-azure-toolkit/configure-the-bootloader-module-for-a-sitecore-deployment.html

I assume that the installjob.exe runs automatically on app start and processes all the files found somehow.

I found the file that actions this (?) as mentioned in another blog post in \App_Config\Sitecore\Azure\Sitecore.Cloud.Integration.Bootload.PostStepsRunner.config

Still not clear how this all ties together.. then I found this helpful post https://kezhan.info/2018/02/26/All-about-SitecoreCloudIntegrationBootload although that is for Sitecore 8 but it sounds like much the same today.

So it sounds like Sitecore Managed Cloud deploy the Bootloader WDP from Sitecore Azure Toolkit into all Sitecore Managed Cloud sites, and this in turn monitors \App_Data\Transforms and installs any package found in there…

Poor man’s approach for diffing your Sitecore 9 database

During an upgrade we needed to compare the core database with the out of the box one to confirm what changes had been made, in case they hadn’t been checked in with Unicorn.

There is a great tool called Razl that I recommend you look at. It used to provide a free trial but not any longer.

As we only wanted to do a quick diff I tried an alternative approach, using the built-in serialisation tools.

  1. Set up a Sitecore instance using the target database
  2. Log into Sitecore as admin
  3. Right click the content editor ribbon and enable Developer
  4. Choose the portion of the tree that you want to diff, click Serialize Tree
  5. Download the contents of the App_Data/Serialization folder (or just the sub folder that represents the part of the tree you are interested in) it might help to clear out the Serialization folder if you aren’t using it and want to just compare it wholesale
  6. Run through steps 1 – 5 using a Sitecore instance pointed at the source version of the database you want to diff (for example, the vanilla Sitecore database for your Sitecore version)
  7. Use a tool such as WinMerge to compare the two folders, and hide the identical items. It is actually quite easy to navigate and diff to see the changes that have been made.
  8. Diff any files that represent items to see what has been changed. If still unclear, use Sitecore to load up the item in question and see the changes in the content editor.

Happy diffing!

Speeding up Sitecore 9.1.1 Experience Editor with Limited Page Editor role

After an upgrade from Sitecore 9.0 to 9.1.1 we found that the Experience Editor load time had gone from 8 seconds to over a minute! The time is all spend loading Ribbon.aspx.

Sitecore have been unable so far to determine the cause of the slowdown, but various posts imply that other people have the same issue either on new or upgraded 9.1 sites, and that a good way to speed this up is to tweak the page editor ribbon options.

I spend some time looking at the built-in roles that might help.

I found that any user that inherits from sitecore\Author is extremely slow (~70 seconds). The tabs include “Optimization” which is not even needed by our site which has xDB Disabled!

It looks like this is due to sitecore\Author inheriting from Analytics roles:

  • sitecore\Author
    • sitecore\Analytics Testing
    • sitecore\Analytics Personalization
    • sitecore\Sitecore Client Authoring

If I use sitecore\Sitecore Client Authoring instead then the load time goes down to 40 seconds and it no longer includes the Optimization tab.

This implies that the Optimization tab was adding about 30 seconds. But 40 seconds is still too slow.

I found that by adding the role sitecore\Sitecore Limited Page Editor in addition to either of the above roles then it loads in 15 seconds but only the ribbon contains 2 tabs – Home and Versions. However the tick boxes on the View tab are present on the Home tab.

The only issue here is that the Add Component button is greyed out. I found that this is due to it being Denied read access in the item in the core database – /sitecore/system/Settings/Security/Policies/Page Editor/Can Design

Once I removed this, the design features are available and the experience editor appears perfectly usable, and much faster.

If / when we find a better way to resolve this without limiting the features, I will update this post!

Add Page Views column to the table on Sitecore Analytics Reports

On many pages of the Sitecore Analytics dashboard, there is a table of visit data.

This by default is sorted by page views descending, but there is no page views column! The Visits column is actually the number of visits where at least one page view hit the page in question.

We were asked to add Page Views to this table, and it turned out to be incredibly easy.

In the Core database, go to /Sitecore/client/Applications/ExperienceAnalytics/Common/System/ListControl and you’ll see all the headings from the above table are items under this one:

 

Simply duplicate the Visits item and change the header to “Page views” and data field to “pageViews” as per the above screenshot.

Hey presto:

Also worth pointing out you can set the default sort option here under the relevant ExperienceAnalyticsListControl Parameters item (see my previous post for more detail on customising the Analytics pages):

 

 

Deploying Sitecore Unicorn Items to Azure Web App using VSTS Build and Release

For years we have been using Unicorn to manage and deploy our developer-owned Sitecore items using TeamCity. Nowadays, we have moved to using VSTS for source code, build and release management, and Microsoft Azure for web app hosting. We struggled to set up a process akin to our TeamCity deployment process and Darren Guy’s epic blog series only covers Octopus Deploy.

I won’t go into setting up Unicorn as this is covered in the documentation. I will assume you have got it saving serialized files into source control and just outline the solution we have used to deploy the items.

We decided to put the Unicorn files into /App_Data/Unicorn so it is beneath the Webroot so can be deployed to, but protected from the public.

1. Configure the build to include Unicorn files

We used a Copy Files step to copy the Unicorn folder from source control into the artefact staging directory. This means the files are available at Release time. We’re copying it into a new folder “UnicornWWW” in the location under the website path that we want to deploy the items to (App_Data/Unicorn).

We also zip this folder up into a Unicorn.zip which contains the path within the website inside the zip, and delete the temporary folder as we don’t need it anymore.

Now the UnicornWWW.zip is a build artefact available at release time. You can check by going to the completed build summary page and checking the Artefacts tab, where the file should be listed.

2. Deploy the Unicorn files

This was tricky as we want to not only deploy the new files but also delete any old ones that have been removed from source control. We already had an Azure App Service Deploy step to deploy our web deploy package to the web app. What we did was add a post deployment script to this step which removes the Unicorn folder:


Then we have a follow up deployment which deploys the UnicornWWW.zip directly to the web app. As it has the App_Data/Unicorn folder inside it, the Unicorn files are deployed to the right place and the previous version have been deleted beforehand.

I believe it should be possible to do a manual msdeploy call inside a Powershell script where you could target a subfolder and do ‘sync’ to delete existing files, much like we do with TeamCity, but we haven’t got that working quite yet as not sure how the publish credentials would be taken out of the service principal setup that connects VSTS and Azure – it “just works” when you use the Azure App Service Deploy task so for now this is what we are sticking with.

3. Sync the Unicorn files

First we need a copy of the Unicorn Powershell API scripts available to the release agent. To do this we go back to the build and add a step to copy the Unicorn Tools PSAPI folder from the nuget package location into the build artefact folder:


We then use a Powershell step which runs on the Release agent, which triggers Unicorn sync using the Unicorn Powershell API. This is based on the sample.ps1 from the PSAPI folder. Here we specify the environment URL + /unicorn.aspx and the shared secret that is configured as per Unicorn.UI.config.


The URL and secret can be made environment variables and then you have a fully automated deployment of Unicorn items to your Azure web app environment!

Customising Sitecore Experience Analytics Dashboard

We had a request to customise the default Experience Analytics dashboard on a Sitecore 8.1 site. With a little guidance from Sitecore we have been able to do this easily without development, but it is a bit of a convoluted process so I thought I would outline it here.

If possible, get your site live with Analytics configured correctly and capturing data well before you start looking at customising it, as you can then browse around the default analytics pages and see the real data that is captured. You can then give an analyst access and ask them what changes would be useful.

The default dashboard looks a little like this, on a site that hasn’t had any Analytics customisation done:

All the blocks apart from are ‘by value’ which means there is no data because no profiles / personas / experience scoring has been configured. Therefore, our analyst asked if the dashboard could be changed to include more common analytics like you might find on a Google Analytics dashboard.

  • Total Visits (line chart)
  • Top Referring Site (bar graph)
  • Top Entry Pages (bar graph)
  • Pageviews + Pageview/Visit (line chart)
  • Top pages (bar chart)
  • Top pages over time (line chart)
  • Top 10 pages (table)

These are all graphs that she found in other Analytics report pages, so all we had to do is figure out how to remove the existing charts from the dashboard and copy these from the other pages, creating a new dashboard layout.

The most relevant Sitecore documentation discusses how to create a new report page with a single chart. Using this and a bit of Sitecore knowledge you can figure out how to edit an existing complex dashboard.

The analytics reports structure is stored as Sitecore items in the core database. You can view it by using Sitecore Desktop (link from the Sitecore launchpad) and switch to Core DB using the link at the bottom right:

Then open Content Editor and browse to /sitecore/client/Applications/ExperienceAnalytics/Dashboard

You can see that beneath Dashboard are all the report folders (Audience, Behaviour, etc – which are just common Folders) and report pages (Overview, Devices, etc – using template /sitecore/client/Speak/Templates/Pages/Speak-DashboardPage)

Each report page has a PageSettings folder (template = /sitecore/client/Speak/Templates/Pages/PageSettings) that contains an item underneath it for every chart on the page. These items allow you to set the chart title and any other settings such as which metrics / dimensions to use on the chart.

Review the Layout Details for any page with multiple charts and you can see it is made up of a series of Rows and Columns which define the page layout, and renderings for each chart.

Click Edit and you can see the placeholder IDs used – now you can see how the page is put together. RowPanels are rendered one above the other (in Main.Content placeholder) and each RowPanel automatically defines a placeholder “RowPanel X.Content” where X is the row number. Then ColumnPanels are inserted into the RowPanel placeholders to set up however many columns you want to break that row into. In this example, one full width row, then two 50/50 rows.

The chart renderings themselves are inserted into the ColumnPanels using autogenerated placeholders based on the row and column number e.g. “ColumnPanel 3.2.Content” means this rendering will be shown in row 3 column 2.

To edit this becomes more interesting because you can’t simply edit this in Sitecore – Edit gives you a server error, and Change gives you an empty list of renderings:

So based on the Sitecore article we have to switch to Sitecore Rocks which is able to edit these. Install and connect to the Sitecore instance, then expand Core database to the same location. Select a report page, Right-click, Tasks, Design Layout:

Now you can edit the configuration of rows/columns by moving/adding/removing RowPanel and ColumnPanels, editing the Placeholder of each. Note that each ColumnPanel has a ‘GridColumns’ field set to 12 for full width, 6 for half width, 4 for 1/3 width, etc. Each RowPanel seems to be used for a single Row, not floating multiple charts into it.

The charts themselves use one of a number of Renderings all starting with ‘ExperienceAnalytics’ – you can copy/paste these using right click options from one page’s layout to another, or Add Rendering to pick one to add from scratch:

Each chart needs to have its Datasource set to a PageSettings child item using the matching parameters template:

Easiest is to find and copy the existing settings item from another report page, under the dashboard page, and then edit the datasource of the chart rendering item to use the new settings item.

Save regularly and reload the dashboard page, to see your changes. Here is my modified dashboard page where I have different charts:

How to shrink a Sitecore database

We had some databases that were 25 gb each due to some large media items.

To shrink the database, it is not as simple as deleting the large items.

After you have identified and deleted any unnecessary items from Sitecore, do this on each database (or publish the parent section) and then Empty the Sitecore recycle bin (in each database, via the Sitecore Desktop menu).

Then you have to run the Sitecore Clean Up Database wizard. This will remove the orphaned items from the DB. Finally you can shrink the databases and the space will actually be reclaimed.

How to migrate users and passwords between Sitecore instances

If you ever package users from one Sitecore instance to another you will find that upon install the users are disabled and the passwords need to be reset.

What if you were moving them from one version to another and need to keep the same passwords?

Fear not, just run this SQL script. It will update the passwords of all matching usernames to the original passwords, and enable the accounts. The databases must be on the same instance.

/******

 SELECT Upgraded Core DB in SQL Studio.

 In script below, Replace Old_Core with name of old DB

 First run the SELECT only to see what it's going to do, then run the update script.

 ******/
SELECT TOP 1000
      m.[UserId],
      om.userid,
      m.[Password],
      om.password,
      m.[PasswordSalt],
      om.passwordsalt,
      u.username,
     ou.username,
      m.comment,
     m.islockedout,
     om.islockedout,
     m.isapproved,
     om.isapproved
  FROM [aspnet_Membership] m
  inner join [aspnet_Users] u ON m.UserID = u.UserID
  inner join Old_Core.dbo.aspnet_Users ou ON ou.username = u.username
  inner join Old_Core.dbo.aspnet_Membership om ON ou.userid = om.userid

   UPDATE [aspnet_Membership]
  SET
  --comment = ou.username,    -- TEST!
  Password = om.Password,
  PasswordSalt = om.PasswordSalt,
  IsLockedOut = om.IsLockedOut,
  IsApproved = om.IsApproved
  FROM [aspnet_Membership] m
  inner join [aspnet_Users] u ON m.UserID = u.UserID
  inner join Old_Core.dbo.aspnet_Users ou ON ou.username = u.username
  inner join Old_Core.dbo.aspnet_Membership om ON ou.userid = om.userid

Sitecore 8 Federated Experience Manager

I recently posted a short introduction to the Sitecore FXM on the e3 blog, below is a more detailed version!

One new feature in Sitecore 8 is the Federated Experience Manager (FXM). This is a brand new tool designed to allow you to apply Sitecore’s marketing features to existing non-Sitecore sites. This would be ideal for clients who are investing in the Sitecore platform but also have legacy sites which aren’t being redeveloped into Sitecore. For example, associated WordPress blogs, Drupal communities, even flat microsites could contribute to a user’s Sitecore Experience profile and affect personalisation across all these sites. I’ve had a chance to experiment with this tool and this is what I found.

This new tool is an option in the new Sitecore Launchpad. It presents you with a list of sites that you have added already, along with number of visits that have been tracked through the FXM:

After creating a federated site they show in this list. For testing purposes I have added the e3 test website as a federated site. You need to enter the site’s domain and then paste some tracking script onto every page of the target site. The details of my federated site are as follows:

Now comes the fun bit. Clicking on “Open in Experience Editor” brings up a new version of the Experience Editor, geared towards being able to edit federated sites. In my instance, here you can see the e3 homepage showing up within the experience editor of my fresh Sitecore 8 instance:

From this visual editor you are able to do various things such as:

  • Attach Sitecore functionality to links / button clicks (such as triggering a goal)
  • Assign goals/campaigns/events to trigger when certain page(s) are viewed
  • Add Sitecore placeholders at any point within the existing site, to inject new content or replace existing content

As a basic example, we could trigger a goal when a user views a case study and then display a personalised block on the homepage to highlight a particular case study, such as this one about Unicef Launchpad.

The first thing you must do is add a Page Filter. This is how Sitecore will target your personalisation to decide which page(s) it affects. Here we create a filter to match all case study pages, and assign a custom “Viewed Case Study” goal I made earlier:


Next, for the personalisation on the Homepage, I can click “Add Placeholder” and then click on a part of the page. This selects the component I have clicked and gives a choice of adding before or after that component:

In this case I have chosen “Add before” as I want to add a new component in above “Our Latest Work”. Once the placeholder is added, I can add any existing Sitecore layout in the new placeholder (for example any design component you already have built for your Sitecore site), and display the desired content. The federated site may need updated CSS to correctly display design elements from the Sitecore site, otherwise they might appear unstyled. Once I’ve finished adding the new component it looks like this:

By default this new content will display all the time. The final step is to apply some personalisation to the new block. To do this, I can select the block and choose Edit personalisation from the floating toolbar:

The following personalisation settings will show the custom content if the user has viewed a case study, otherwise hide the component. You could add multiple conditions in here to display different content depending on the user’s activity.

Once all this is done and published, I can browse to the target site and it appears as normal. If I browse to any case study and then back to the homepage, I can see the new component has been injected into the page’s content:

There are many possibilities in terms of the personalisation logic you could apply, and this new ability to apply marketing features to external sites really adds a new dimension to Sitecore’s experience platform.

Automated Site Setup Scripts (Sitecore / Umbraco)

We wanted a simple way to get a new environment set up on a developer machine, but without the hassle of Virtual Machines.

Enter, Microsoft’s appcmd.

This lets you manipulate IIS7 via command line calls. There is lots of documentation on the IIS site but it takes a while to figure out which commands you need to do everything to set up a site. So I’ve put together a selection of the useful commands below in a sample site setup script.

We save these as batch files (.bat) and put them in a shared folder.

Prerequisites:

  • You work on Windows using IIS7
  • (if Sitecore) You have all relevant Sitecore installation zips in a shared network folder
  • (if Umbraco) You use Nuget package to distribute Umbraco CMS (or you could do the zip thing)
  • You run sites locally with custom host name, using a shared DB
  • All team members setup sites in the same folder path e.g. c:\work
  • You have admin rights on the local machine

Sample setup script:

Put this in a .bat file and save in a shared folder. Keep as many steps as required, and replace the relevant placeholders. To run, right click Run as Administrator.

@ECHO OFF

ECHO *** MUST BE RUN AS ADMIN - IF NOT PLEASE CLOSE AND RUN AGAIN, OTHERWISE: ***
PAUSE

ECHO *** CHECKING OUT FROM SVN ***

IF not exist c:\PATH_TO_TRUNK svn checkout -q https://SVN_PATH_TO_TRUNK c:\PATH_TO_TRUNK

ECHO *** EXTRACTING SITECORE ***

"C:\Program Files\7-Zip\7z" x -y "PATH_TO_SITECORE_ZIPS\Sitecore 7.0 rev. 140120.zip" -oc:\PATH_TO_TRUNK "Sitecore 7.0 rev. 140120\Data" "Sitecore 7.0 rev. 140120\Website"

ECHO *** RENAMING SITECORE ***

rename "c:\PATH_TO_TRUNK\Sitecore 7.0 rev. 140120" www

ECHO *** COPY LICENSE ***

copy PATH_TO_LICENSE\license.xml c:\PATH_TO_TRUNK\www\Data

ECHO *** CREATING APP POOL ***

C:\Windows\System32\inetsrv\appcmd add apppool /name:"APP_POOL_NAME" -managedRuntimeVersion:v4.0

ECHO *** CREATING SITE ***

C:\Windows\System32\inetsrv\appcmd add site /name:"APP_POOL_NAME" /bindings:http://site.localhost:80 /physicalPath:"c:\PATH_TO_TRUNK\www\Website" -applicationDefaults.applicationPool:APP_POOL_NAME

ECHO *** CREATING VIRTUAL DIRECTORIES ***

C:\Windows\System32\inetsrv\appcmd add vdir /app.name:"APP_POOL_NAME/" /path:/siteAssets /physicalPath:c:\PATH_TO_TRUNK\siteAssets

ECHO *** ADDING TO HOSTS FILE ***

find /c "	site.localhost" c:\windows\system32\drivers\etc\hosts
if %errorlevel% equ 1 goto notfound
echo Already there
goto done
:notfound
ECHO 127.0.0.1	site.localhost >> c:\windows\system32\drivers\etc\hosts
goto done
:done
  
ECHO *** BUILDING SOLUTION ***

C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe c:\PATH_TO_TRUNK\src\Site.sln /verbosity:quiet

ECHO *** OPENING SITE ***

start http://site.localhost/

PAUSE

Hope that’s helpful to someone!