I recently published an article on how to set up a file Impact test For the ubiquitous “flicker effect” in client-side A/B testing tools. Be sure to check out this article first to get some context on what we’ll be talking about here.
In this short follow up, I’ll show you how to measure the average anti-flicker snippet delay time, if you choose to publish the snippet. The methodology is very similar.
If you remember, the anti-flicker extract hiding The entire page is waiting for the Optimize container to load. So we’ll measure how long the page took to become unhidden. Visibility is restored if the Optimize container loads successfully, or if the loading finishes in a timeout (4 seconds by default).
The test below is done by splitting 50% of the traffic to the asynchronous Optimize snippet and 50% of the traffic to the Google Tag Manager Optimize tag.
We are using Google Analytics: App + Web with a great BigQuery export for the analysis. We will use Google Tag Manager to collect data and send it forward.
Simmer . Newsletter
Subscribe to the Simmer newsletter to get the latest news and content from Simo Ahava right in your inbox!
Edit the page template
You need to edit a file page template. Anti-flicker extract Should To be added directly to the page template, we also need to write the logic that defines whether the user should see the Optimize snippet or whether Optimize should be loaded via Google Tag Manager.
In itself higher subordinate
<head> On your experiment pages, add the following HTML block:
<!-- The style declaration for the anti-flicker snippet --> <style>.async-hide opacity: 0 !important </style> <script> (function() // Modify the optimizeId to match your Optimize container ID, gtmId // to match your GTM container ID, and dataLayerName to match the name // of the dataLayer array on your site. var optimizeId = 'GTM-NGM64B', gtmId = 'GTM-PZ7GMV9', dataLayerName = 'dataLayer', hideObj = , hideGTMId = Math.random() < 0.5 ? optimizeId : gtmId; hideObj[hideGTMId] = true; // Helper to handle the dataLayer.push() var dPush = function(status) window[dataLayerName].push( event: 'optimize_anti_flicker_test', milestones: antiFlickerStart: window[dataLayerName].hide.start, antiFlickerEnd: new Date().getTime(), testStatus: status ); ; // MODIFIED anti-flicker snippet (function(a,s,y,n,c,h,i,d,e) ).hide = h; var t = setTimeout(function() dPush('timeout'); i(); h.end = null; , c); h.timeout = c; )(window, document.documentElement, 'async-hide', dataLayerName, 4000, hideObj); // Determine where to load Optimize from (inline vs. GTM) if (hideGTMId === optimizeId) var el = document.createElement('script'); el.src = 'https://www.googleoptimize.com/optimize.js?id=' + optimizeId; el.addEventListener('error', function() dPush('optimizeSnippetError'); window[dataLayerName].hide.end && window[dataLayerName].hide.end(); ); document.head.appendChild(el); else window[dataLayerName].push( gtmOptimize: true ); // Configure the Optimize callback function gtag() dataLayer.push(arguments); gtag('event', 'optimize.callback', callback: function() dPush(hideGTMId === optimizeId ? 'optimizeSnippet' : 'gtmTag'); ); )(); </script>
You should only add this snippet on pages that are already running the experiment, to make sure you’re not accidentally collecting measurements from pages that aren’t already running Optimize.
In the first set of variables, be sure to update
dataLayerName Reflects the optimization ID, the Google Tag Manager container ID, and a file name
dataLayer group, respectively.
var hideGTMId = Math.random() < 0.5 ? optimizeId : gtmId; It randomly chooses (50% chance) whether to load Optimize using the asynchronous inline snippet or through the Google Tag Manager tag.
The Anti-flicker extract He is Modified in this version. The main change is that if Timeout (by default 4000 ms after hiding the page), a
dataLayer.push() with this information. For this reason, another edit to the snippet is Stop Timeout counter if the page is not hidden (to avoid misreporting the timeout
h.end = i = function() clearTimeout ... var t = setTimeout(function() dPush('timeout'); ... , c);
The following block checks whether Optimize should be loaded via the snippet or via GTM.
// Determine where to load Optimize from (inline vs. GTM) if (hideGTMId === optimizeId) var el = document.createElement('script'); el.src = 'https://www.googleoptimize.com/optimize.js?id=' + optimizeId; el.addEventListener('error', function() dPush('optimizeSnippetError'); window[dataLayerName].hide.end && window[dataLayerName].hide.end(); ); document.head.appendChild(el); else window[dataLayerName].push( gtmOptimize: true );
If the inline snippet wins the drawing, the optimization element is added to the page with a file
error A listener that detects the page if an Optimize loading error occurs (eg user blocks script loading).
If Optimize is loaded via Google Tag Manager, then the key
gtmOptimize pushed to
dataLayer with value
true. This is then used later as a triggering condition for the Optimize tag.
Once the Optimize container or Google Tag Manager is loaded, the anti-flashing snippet times out, or there is an error loading Optimize,
dataLayer.push() It happens with the following content:
event: 'optimize_anti_flicker_test', milestones: antiFlickerStart: window[dataLayerName].hide.start, antiFlickerEnd: new Date().getTime(), testStatus: status
status is one of the
gtmTag (If Optimize is loaded via Google Tag Manager),
optimizeSnippet (if uploaded via Optimize),
timeout (if the timeout is reached), or
optimizeSnippetError (If an error occurs in the Optimize snippet).
One thing to note is that this setting does Not Test if Google Tag Manager is blocked. This is something you might want to test as well, if you want to get a more comprehensive idea of what’s going on in your experiment apps.
Setting up Google Tag Manager
In Google Tag Manager, we will need to create a file app + web tag (Because we want to do the analysis in BigQuery again). We will also need a file Custom event trigger and some data layer variables.
This is what the “custom event” trigger looks like.
This trigger will fire whenever a file is
dataLayer.push() With test data snippet on the page. There is also a requirement to activate this tag only on the home page, which you can and should edit/remove if you are experimenting elsewhere too!.
you will need four
|variable name||Data layer variable name field value||purpose|
|DLV – Landmarks||
|DLV – Milestones||
||The timestamp of when the page was hidden.|
|DLV – Milestones||
||Timestamp for when the page has ended.|
|DLV – gtmOptimize||
This is what the variable would look like, as per the specifications in the table above:
First, you will need to create a file App tag + web event. Make sure you have a basic tag too!
The flag is set to be triggered using the Custom Event Trigger we created above, and sends the values of the three
milestones Application + web variables as custom parameters. Feel free to change the keys and event names as you wish.
Next, we need to launch the Google Optimize tag on terms, depending on whether
gtmOptimize key in
dataLayer with value
Since Optimize needs to work in a sequence of tags, this is very complicated. In addition to the Google Optimize tag itself, you need a file View the Universal Analytics page The tag that activates only when
gtmOptimize He is
true, you need to block normal page view tag In this circumstance as well.
Don’t worry about setting up the tag sequence, you are setting it up in the new page view tag which you will also need to create:
Note the new launcher: “All Pages – Anti-flicker” is a file page width Trigger with one condition:
DLV - gtmOptimize equals true.
Finally, make sure you block is yours Normal Pageview flag to avoid double counting on pages where optimization’s pageview is already activated:
The block trigger is a custom event trigger that blocks any event if
gtmOptimize He is
Once everything is set up, try loading the page by modifying the page template in preview mode. Make sure you see a request for the app + web with your custom parameters in place.
If it does not work, check for errors in the browser console. Also, make sure that the Optimize container is already loaded, and that you have the trial running on the page you’re testing on!
Dig deep with BigQuery
Once the data starts flowing into BigQuery, you should find your events using a query like this:
SELECT * FROM `project.dataset.events_202006*` WHERE event_name = 'optimize_anti_flicker_snippet_test'
We simply load all visits with the . extension
optimize_anti_flicker_snippet_test data for an overview of what these results look like.
To get the number of different types of tests, you can run a query like this:
SELECT (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'test_status') as test_status, COUNT(*) as count FROM `project.dataset.events_202006*` WHERE event_name = 'optimize_anti_flicker_snippet_test' GROUP BY 1 ORDER BY 2 DESC
This query pulls a value
test_status of events, and makes a total count for each case. This is what the end result will look like:
Finally, to get some averages in place, you can modify the query to look like this:
WITH milestones AS ( SELECT (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'test_status') as test_status, (SELECT value.int_value FROM UNNEST(event_params) WHERE key = 'anti_flicker_start') as anti_flicker_start, (SELECT value.int_value FROM UNNEST(event_params) WHERE key = 'anti_flicker_end') as anti_flicker_end FROM `project.dataset.events_202006*` WHERE event_name = 'optimize_anti_flicker_snippet_test' ) SELECT test_status, COUNT(*) as count, ROUND(AVG(anti_flicker_end - anti_flicker_start), 2) as average_delay_in_ms, FROM milestones WHERE anti_flicker_end - anti_flicker_start < 5000 GROUP BY 1 ORDER BY 3 DESC
WITH...AS The block creates a source table with only the test case and flash start and end times in place. We can then query the Common Table Expression (CTE) to get the counts and averages correctly.
As you can see, I have a file
WHERE Clause in place where I make sure deltas are no more than 5000 milliseconds. This is because sometimes the experiment resulted in abnormally high delta rates, possibly due to the presence of the Optimize container until far away Slow to load, thus producing an expiry time much later than a timeout.
WHERE Condition, we ignore such abnormal deltas. We can do this because we are only interested in how long the page will last hidden, and if the page is hidden for more than 4 seconds (and changed), it will be automatically detected by the anti-flicker snippet itself.
Anyway, this query yields the following result for my dataset:
The average time to hide the page if the snippet times out is 3.8 seconds. This is kind of weird because the timeout shouldn’t have happened before
4000 A millisecond has passed. I was going to dig deeper into it, but I can also assume that’s all
timeout Duplicates were hidden within the full 4 seconds.
When loading Optimize using the Google Tag Manager tag, the page is hidden with an average of 964ms. When loaded with the inline asynchronous snippet, it caches for an average of 581ms.
It’s a fairly complicated setup, but the purpose is not to do this for every one of your experiments. So to satisfy your curiosity about whether or not the anti-flicker snippet degrades the user experience, this just gives you One Variable to work with (average delay page-hide).
It’s not without its flaws – the timer starts when anti-flicker snippet is executed at the top
<head>, when you should start realistically only when the page will normally Produce The first visible element (First fee guaranteed).
The analysis is separate as well – just knowing the delay is not very interesting. What makes it even more important is knowing if the length of the anti-flicker effect has an effect on how the user interacts with the site. A long delay in painting the page is likely to cause the user to return to the previous page, as they may assume that the page is not working.
Just to make the point clear: This article showed you a methodology that you can use to measure the impact of flicker mitigation efforts in client-side A/B testing. blink He is problem and find out how Many The problem is the first step in solving it.
Let me know in the comments if you have suggestions to improve the experience!