Google Optimize Anti-flicker Snippet Delay Test – News Couple
ANALYTICS

Google Optimize Anti-flicker Snippet Delay Test


I recently published an article on how to set up a file Impact test For the ubiquitous “flicker effect” in client-side A/B testing tools. Be sure to check out this article first to get some context on what we’ll be talking about here.

In this short follow up, I’ll show you how to measure the average anti-flicker snippet delay time, if you choose to publish the snippet. The methodology is very similar.

If you remember, the anti-flicker extract hiding The entire page is waiting for the Optimize container to load. So we’ll measure how long the page took to become unhidden. Visibility is restored if the Optimize container loads successfully, or if the loading finishes in a timeout (4 seconds by default).

The test below is done by splitting 50% of the traffic to the asynchronous Optimize snippet and 50% of the traffic to the Google Tag Manager Optimize tag.

We are using Google Analytics: App + Web with a great BigQuery export for the analysis. We will use Google Tag Manager to collect data and send it forward.


X


Simmer . Newsletter

Subscribe to the Simmer newsletter to get the latest news and content from Simo Ahava right in your inbox!

Edit the page template

You need to edit a file page template. Anti-flicker extract Should To be added directly to the page template, we also need to write the logic that defines whether the user should see the Optimize snippet or whether Optimize should be loaded via Google Tag Manager.

In itself higher subordinate <head> On your experiment pages, add the following HTML block:

<!-- The style declaration for the anti-flicker snippet -->
<style>.async-hide  opacity: 0 !important </style>

<script>
  (function() 
    // Modify the optimizeId to match your Optimize container ID, gtmId
    // to match your GTM container ID, and dataLayerName to match the name
    // of the dataLayer array on your site.
    var optimizeId = 'GTM-NGM64B',
        gtmId = 'GTM-PZ7GMV9',
        dataLayerName = 'dataLayer',
        hideObj = ,
        hideGTMId = Math.random() < 0.5 ? optimizeId : gtmId;
    
    hideObj[hideGTMId] = true;
    
    // Helper to handle the dataLayer.push()
    var dPush = function(status) 
      window[dataLayerName].push(
        event: 'optimize_anti_flicker_test',
        milestones: 
          antiFlickerStart: window[dataLayerName].hide.start,
          antiFlickerEnd: new Date().getTime(),
          testStatus: status
        
      );
    ;

    // MODIFIED anti-flicker snippet
    (function(a,s,y,n,c,h,i,d,e)  []).hide = h;
      var t = setTimeout(function() 
        dPush('timeout');
        i();
        h.end = null;
      , c);
      h.timeout = c;
    )(window, document.documentElement, 'async-hide', dataLayerName, 4000, hideObj);

    // Determine where to load Optimize from (inline vs. GTM)
    if (hideGTMId === optimizeId) 
      var el = document.createElement('script');
      el.src = 'https://www.googleoptimize.com/optimize.js?id=' + optimizeId;
      el.addEventListener('error', function() 
        dPush('optimizeSnippetError');
        window[dataLayerName].hide.end && window[dataLayerName].hide.end();
      );
      document.head.appendChild(el);
     else 
      window[dataLayerName].push(
        gtmOptimize: true
      );
    
    
    // Configure the Optimize callback
    function gtag() dataLayer.push(arguments);
    gtag('event', 'optimize.callback', 
      callback: function() 
        dPush(hideGTMId === optimizeId ? 'optimizeSnippet' : 'gtmTag');
      
    );
  )();
</script>

You should only add this snippet on pages that are already running the experiment, to make sure you’re not accidentally collecting measurements from pages that aren’t already running Optimize.

In the first set of variables, be sure to update optimizeIdAnd gtmId, And dataLayerName Reflects the optimization ID, the Google Tag Manager container ID, and a file name dataLayer group, respectively.

var hideGTMId = Math.random() < 0.5 ? optimizeId : gtmId; It randomly chooses (50% chance) whether to load Optimize using the asynchronous inline snippet or through the Google Tag Manager tag.

The Anti-flicker extract He is Modified in this version. The main change is that if Timeout (by default 4000 ms after hiding the page), a dataLayer.push() with this information. For this reason, another edit to the snippet is Stop Timeout counter if the page is not hidden (to avoid misreporting the timeout dataLayer):

h.end = i = function() 
  clearTimeout
  ...

var t = setTimeout(function() 
  dPush('timeout');
  ...
, c);

The following block checks whether Optimize should be loaded via the snippet or via GTM.

// Determine where to load Optimize from (inline vs. GTM)
if (hideGTMId === optimizeId) 
  var el = document.createElement('script');
  el.src = 'https://www.googleoptimize.com/optimize.js?id=' + optimizeId;
  el.addEventListener('error', function() 
  dPush('optimizeSnippetError');
    window[dataLayerName].hide.end && window[dataLayerName].hide.end();
  );
  document.head.appendChild(el);
 else 
  window[dataLayerName].push(
    gtmOptimize: true
  );

If the inline snippet wins the drawing, the optimization element is added to the page with a file error A listener that detects the page if an Optimize loading error occurs (eg user blocks script loading).

If Optimize is loaded via Google Tag Manager, then the key gtmOptimize pushed to dataLayer with value true. This is then used later as a triggering condition for the Optimize tag.

Once the Optimize container or Google Tag Manager is loaded, the anti-flashing snippet times out, or there is an error loading Optimize, dataLayer.push() It happens with the following content:


  event: 'optimize_anti_flicker_test',
  milestones: 
    antiFlickerStart: window[dataLayerName].hide.start,
    antiFlickerEnd: new Date().getTime(),
    testStatus: status
  

Here, status is one of the gtmTag (If Optimize is loaded via Google Tag Manager), optimizeSnippet (if uploaded via Optimize), timeout (if the timeout is reached), or optimizeSnippetError (If an error occurs in the Optimize snippet).

One thing to note is that this setting does Not Test if Google Tag Manager is blocked. This is something you might want to test as well, if you want to get a more comprehensive idea of ​​what’s going on in your experiment apps.

Setting up Google Tag Manager

In Google Tag Manager, we will need to create a file app + web tag (Because we want to do the analysis in BigQuery again). We will also need a file Custom event trigger and some data layer variables.

trigger

This is what the “custom event” trigger looks like.

Custom event trigger

This trigger will fire whenever a file is dataLayer.push() With test data snippet on the page. There is also a requirement to activate this tag only on the home page, which you can and should edit/remove if you are experimenting elsewhere too!.

Variables

you will need four dataLayer variables.

variable name Data layer variable name field value purpose
DLV – Landmarks milestones.testStatus one of gtmTagAnd optimizeSnippetAnd timeout, or optimizeSnippetError.
DLV – Milestones milestones.antiFlickerStart The timestamp of when the page was hidden.
DLV – Milestones milestones.antiFlickerEnd Timestamp for when the page has ended.
DLV – gtmOptimize gtmOptimize He is true If Optimize must be loaded into the GTM tag.

This is what the variable would look like, as per the specifications in the table above:

data layer variable

Tags

First, you will need to create a file App tag + web event. Make sure you have a basic tag too!

App + Web Anti-flicker event tag

The flag is set to be triggered using the Custom Event Trigger we created above, and sends the values ​​of the three milestones Application + web variables as custom parameters. Feel free to change the keys and event names as you wish.

Next, we need to launch the Google Optimize tag on terms, depending on whether gtmOptimize key in dataLayer with value true.

Since Optimize needs to work in a sequence of tags, this is very complicated. In addition to the Google Optimize tag itself, you need a file View the Universal Analytics page The tag that activates only when gtmOptimize He is true, you need to block normal page view tag In this circumstance as well.

tag improvement

Don’t worry about setting up the tag sequence, you are setting it up in the new page view tag which you will also need to create:

New page view tag

Note the new launcher: “All Pages – Anti-flicker” is a file page width Trigger with one condition: DLV - gtmOptimize equals true.

Finally, make sure you block is yours Normal Pageview flag to avoid double counting on pages where optimization’s pageview is already activated:

Page view player

The block trigger is a custom event trigger that blocks any event if gtmOptimize He is true:

Ban in case of improvement

Test setup

Once everything is set up, try loading the page by modifying the page template in preview mode. Make sure you see a request for the app + web with your custom parameters in place.

App + web works

If it does not work, check for errors in the browser console. Also, make sure that the Optimize container is already loaded, and that you have the trial running on the page you’re testing on!

Dig deep with BigQuery

Once the data starts flowing into BigQuery, you should find your events using a query like this:

SELECT
  *
FROM 
  `project.dataset.events_202006*`
WHERE
  event_name = 'optimize_anti_flicker_snippet_test'

We simply load all visits with the . extension optimize_anti_flicker_snippet_test data for an overview of what these results look like.

Anti-flicker events

To get the number of different types of tests, you can run a query like this:

SELECT
  (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'test_status') as test_status,
  COUNT(*) as count
FROM
  `project.dataset.events_202006*`
WHERE
  event_name = 'optimize_anti_flicker_snippet_test'
GROUP BY 1
ORDER BY 2 DESC

This query pulls a value test_status of events, and makes a total count for each case. This is what the end result will look like:

Calculates the test condition

Finally, to get some averages in place, you can modify the query to look like this:

WITH milestones AS (
  SELECT 
    (SELECT value.string_value FROM UNNEST(event_params) WHERE key = 'test_status') as test_status,
    (SELECT value.int_value FROM UNNEST(event_params) WHERE key = 'anti_flicker_start') as anti_flicker_start,
    (SELECT value.int_value FROM UNNEST(event_params) WHERE key = 'anti_flicker_end') as anti_flicker_end
  FROM 
    `project.dataset.events_202006*`
  WHERE
    event_name = 'optimize_anti_flicker_snippet_test'
)
SELECT 
  test_status,
  COUNT(*) as count,
  ROUND(AVG(anti_flicker_end - anti_flicker_start), 2) as average_delay_in_ms,
FROM 
  milestones
WHERE anti_flicker_end - anti_flicker_start < 5000
GROUP BY 1
ORDER BY 3 DESC

The WITH...AS The block creates a source table with only the test case and flash start and end times in place. We can then query the Common Table Expression (CTE) to get the counts and averages correctly.

As you can see, I have a file WHERE Clause in place where I make sure deltas are no more than 5000 milliseconds. This is because sometimes the experiment resulted in abnormally high delta rates, possibly due to the presence of the Optimize container until far away Slow to load, thus producing an expiry time much later than a timeout.

with the WHERE Condition, we ignore such abnormal deltas. We can do this because we are only interested in how long the page will last hidden, and if the page is hidden for more than 4 seconds (and changed), it will be automatically detected by the anti-flicker snippet itself.

Anyway, this query yields the following result for my dataset:

Anti-flicker extract test

The average time to hide the page if the snippet times out is 3.8 seconds. This is kind of weird because the timeout shouldn’t have happened before 4000 A millisecond has passed. I was going to dig deeper into it, but I can also assume that’s all timeout Duplicates were hidden within the full 4 seconds.

When loading Optimize using the Google Tag Manager tag, the page is hidden with an average of 964ms. When loaded with the inline asynchronous snippet, it caches for an average of 581ms.

summary

It’s a fairly complicated setup, but the purpose is not to do this for every one of your experiments. So to satisfy your curiosity about whether or not the anti-flicker snippet degrades the user experience, this just gives you One Variable to work with (average delay page-hide).

It’s not without its flaws – the timer starts when anti-flicker snippet is executed at the top <head>, when you should start realistically only when the page will normally Produce The first visible element (First fee guaranteed).

The analysis is separate as well – just knowing the delay is not very interesting. What makes it even more important is knowing if the length of the anti-flicker effect has an effect on how the user interacts with the site. A long delay in painting the page is likely to cause the user to return to the previous page, as they may assume that the page is not working.

Just to make the point clear: This article showed you a methodology that you can use to measure the impact of flicker mitigation efforts in client-side A/B testing. blink He is problem and find out how Many The problem is the first step in solving it.

Let me know in the comments if you have suggestions to improve the experience!



Source link

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button