A Reply To Dr Biks on $SAVA PTau-181

Data by BruinAnteater @BruinStocks - Twitter, @BruinAnteaterStock - Stocktwits A Quick Introduction

Hello, allow me to introduce myself. On twitter I am known as @BruinStocks, and @BruinAnteaterStocks on Stocktwits. On Benzinga I usually go by BruinAnteater or simply as “Hector”

Due to the hostile nature of controversial arguments on the webz, I will bowout of completely revealing my identity. So mark that strike one for me. I also have disclosed that yes, I am an investor in $SAVA. That should be strike two! Oh, here’s strike three: I am NOT a neurologist, a biochemist, or have any professional training in any aspect of medicine. I am a mathematician, and a pure mathematician at that. I learned stats the first semester I had to teach it at a university. I fell in love with the subject and exclusively teach stats now.

The awesome part though is that I am completely open (except for my ID)! I am giving everyone access to my data sets, the method by which I extracted the data, and the R code I used to analyze the data. You may use that to verify my work or dispute it. It’s all fair game! It does not matter if you think I’m a shady, big-whale, crank or if you think I’m the smartest dude ever, I’ll just let my methods, results and openness speak for themselves. Everything I present here (except for opinions) is EXACTLY what you will get if you run my R code, and use the data sets I provided.  I stand by my work, and it should stand on its own, regardless of who I am, what I know, or how invested I am in the company.

The Controversy

A few weeks ago, Dr Bik (@microbiomDigest) wrote an article that proved some to be quite damning to Cassava Sciences, Inc. SAVA reputation. In this article, Dr. Bik takes particular issue with $SAVA’s data presentation on the effect of simufilam on the biomarker P-Tau 181 as shown in Figure 1.

Dr. Bik did a cross comparison between this graph and a spaghetti plot meant to convey the exact same data. As we can clearly see, Dr. Bik is correct: for some reason a data point belonging to the 100mg experimental group was placed in the data set of the placebo-control group.

My analysis does not include an analysis that includes the outlier in the placebo-group. There is no need for this as $SAVA has acknowledged it’s an error, and no one disputes that the value does not belong in the placebo group. In addition to the outlier appearing in the incorrect data set, Dr. Bik also correctly identified an extra data point in the data, with the outlier she identified 52 data points.

Remi Barbier, $SAVA’s CEO stated in a recent conference call that the outlier was dropped from the data set all together, something that, on-its face sounds like clear data manipulation. I will discuss later why in my opinion this isn’t a big deal.

Nevertheless, the core of this article is to consider the data with the outlier correctly placed in the 100mg group and consider the data with the outlier removed. I present both arguments, so you need not rely on my opinion to make up your own mind.

Initial Attempt at Analyzing Dr. Bik’s Displayed Data

I attempted to use the data that Dr. Bik provided, and did a quick analysis of mean, standard deviation, and error to build a 95% Confidence Interval. What’s a confidence interval? It’s an incredible tool that allows you to infer a likely range of true population parameter (means, proportions, standard deviations, etc) from a sample’s statistics. Why do we do this? Populations are often hard to completely study as they tend be impossibly large in many cases. It’s the reason the Census is only conducted every 10 years; although ideally a census should provide data for every member of the population, the Census will miss a part of the U.S. Population. So normally, scientists must rely on sample data to generate a (hopefully small) range of values (the confidence interval) where the true population parameter will likely be found (no guarantees, we are just 95% sure about it though).

Shown in figure 2 is my short-sighted, amateur hour Excel analysis of Dr. Bik’s data

I won’t dive too deeply into this part of the analysis: Dr. Bik openly admits she isn’t showing all the data because “graphs were too muddled.” So, if you understand what those numbers represent, awesome! Interesting to look at, but unfortunately meaningless since this does not count all 52 data points available.

But we do have a serious issue: we need complete data, and we hope we figure out a way to extract it.

The only fair way to do so is to use the same methodology Dr. Bik uses to extract the data (in science, using a peer’s methods when evaluating their work is the essence behind the concept of repeatability in science). Dr. Bik used the website  WebPlotDigitizer - Copyright 2010-2021 Ankit Rohatgi (automeris.io) to extract data points. So, I went ahead and did the same thing with $SAVA’s plot:

Using this method, I was able to extract the original 52 data points shown in the plot. Before you say it: yes, not all my circles are dead center with the plot points. Did the best I could, the difference between my numbers and Dr. Bik’s are tiny, however.

I have those numbers in two different csv files: one file, which is called PTauBirkCorrection.csv (my apologies to Dr. Bik for misspelling her name on my files); this file has all of the data points in their correct groups, including that outlier that I transferred to the 100mg group, and contains all 52 data points. The second file, named PTauOutlierRemoved.csv contain 51 data point since I eliminated the outlier flat out from the data set.

The data was fed into an R-script for the plotting and data analysis. The file name to my script is Sava9moAnalysis.R (again, my apologies for the name of the file, this is PTau data for the placebo controlled, double blind study of simufilam’s effect on the biomarker, as part of their SavaDX presentation).

Links for all of the files will be provided at the very end.

Results from Analysis in R

For both data sets I did the following:

• Calculated the mean, standard deviation and Error value for a 95% Confidence Interval using the t-statistic.
• Created a plot of the data (just like $SAVA did) and included the confidence intervals within the plot.
• Why a confidence interval? Just makes it easy to see where the intervals overlap
• So what if they overlap? So what? Maaaang, time to take my stats course! When overlaps occur, there is chance for ‘no difference in effect’ to be the case.
• Don’t worry, I also ran ANOVA (analysis of variance). What the hell is that? In short, it tells us how likely ‘no effect’ is happening when generalizing the results to the population. The lower the percent, the less likely there is to be no true effect from simulifam on PTau 181.

So lets go through each analysis. First, I will defer to Dr. Bik’s assertion that no data point should be excluded, no matter how outlandish of a value it may be. This is not necessarily a “wrong thing.” In general, I would completely agree with Dr. Bik. I will discuss my actual opinion/justification for why there is no harm, and in fact more accurate to leave the outlier lying outside.

One of my major points of contention with Dr. Bik’s interpretation was the lack of discussion of what the data would have looked like If we considered the outlier in 100mg. I directly asked Dr. Bik to do this analysis, and to give, at least, a qualitative evaluation of the variability in the data. She never responded.

I wanted to visualize the data as Dr. Bik said it should be done, to make it clear that most of the data points for the 100mg group still huddle around a certain area. An even stronger case can be made for the 50mg group as it never had any outlier. Its critical to also note how much more spread out the placebo’s group is in general.

Why does this qualitive analysis matter? Because it mirrors the nature of AZ as a disease: it’s a highly variable disease with respect to person to person, and it’s highly variable within an individual’s own progression which would include big swings in biomarker data and severity in symptoms of the disease. We expect to see large variances in an Alzheimer’s patient’s biomarker data. What most drugs have failed to show is an ability to make the disease less variable. In other words, other drugs have not shown a consistent efficacy. $SAVA’s data, with or without the outlier, isn’t proving it can cure AZ. It’s visually clear (especially with the uncontested 50mg data) that, at least up to the 28-day point, that simulifam is showing the diseases variability is lessened with use of the drug.

But let’s see some hard numbers. Expectedly, the confidence intervals are large (in other words: less specific) with that outlier in the 100mg group. What sort of numbers are we talking about? Below is a summary of the relevant statistics for this data set:

If you add up the N values (how many data points were in each group) you will notice that it adds up to 52, so indeed I am including all the data shown in $SAVA’s plot. The mean difference in PTau levels were a reduction by 7.53% for the 100mg group, a reduction of 15.03% for the 50mg group, and an increase by 13.71% in the placebo group. Although that sounds impressive (at least it does to me), the truth lies in the standard deviation: the larger that number is the more your data varies from its own mean. In other words: the higher the standard deviation is, the less meaningful the mean becomes. It’s no longer a measure of typicalness. The standard deviation of the 100mg is insane. To be fair, the standard deviation for all groups is downright scary, or rather they make this whole analysis pointless. We can’t tell from a pure numbers perspective if there is anything going on. But visually we can see much more tightness on the majority of the 100mg data, so what gives?

Another way to evaluate the data is to use ANOVA (analysis of variance) which essentially evaluates the likelihood that there is no effect:

• If our current working hypothesis is that there is no true effect on PTau via simulifam, then we should expect the true population mean for each group to be same, or said in stats speak:
  
  in real life.
• One alternative hypothesis is that the is a difference in effect. If that was true, then the means would be different. In stats speak this is: 
  
  in real life.

Here we see our P-value is 0.121. But our significance level is 5%. In order to flat out reject the null, I need my p-value to be less than 0.05. Although you can technically choose any significance level you want, it’s industry standard to choose 5%; sometimes it’s reasonable to choose a 10% SL, but even then, we would fail to reject the null.

Let me be clear: just because my data can’t satisfactorily deny the null hypothesis, it does not imply the null hypothesis is correct. I just don’t have enough data to say its likely wrong.

Keeping the outlier in the 100mg group does indeed tie our hands up a bit. What would the data look like if we removed the outlier from the data set flat out? Let’s look:

Figure 5 - PTau 181 change with outlier from 100mg removed

Wow, what a difference! Look at them confidence intervals! Look how much tighter 100mg is now! We still have some overlap between 100mg and placebo but it’s tiny!

Figure 6 - Summary of Statistics with no outlier in 100mg

The stats now show there is an average reduction in the 100mg by 16.8%, with a still massive but is much smaller standard deviation of 35.4. In this situation the coefficient of variance (% wise the ratio of standard deviation to the mean) is about 200%, which is still massive, is much smaller than the placebo’s 350%! In other words: we can almost say the 100mg group has less variance than the placebo group. The 95% confidence interval is -35% to 1.4%. In other words, I am 95% sure that the true mean effect of 100mg of simufilan in PTau level to between a 35% reduction to a 1.4% increase. Yes, that does leave a tiny bit of room for there to be “no effect.” Overall, though, this is an exciting result!

Running ANOVA with a 5% significance level, we get:

Figure 7 - ANOVA on data sets with no outlier

If I haven’t stated this before, let me be clear now: this doesn’t prove that simufilam will effectively treat ALZ, but it does give us solid support for the claim that simufilam does affect PTau181 in a way that is unlike placebo.

Opinion: Why is it OK to throw away the outlier?

What gives me or Remi the right to toss out an outlier? From a pure statistical point of view, the whole point of doing analysis on samples is to see how the data might impact the “middle majority” of patients. Ideally, we’d like to have a strong effect on as many patients as we can, but no treatment for any disease is ever guaranteed to work. The goal is to be effective for most of the population, but we know extreme cases will likely need other treatment.

I hate making analogies because they are a logical no-no, but sometimes they can help illustrate a concept because they provide a more relatable example, so I will do one here. Suppose I randomly ask 30 instructors at my campus to report their net worth. We will certainly get a variety of answers. Some might be fresh out of grad school and buried in debt, the majorities’ net worth will likely be higher than $1M, and a few, especially the veteran department chairs will likely have a few million dollars of net worth. If I find this average, its likely to reflect the true average net worth of the university faculty. Now imagine that Bill Gates serves as guest lecturer at the same time I’m doing my sampling, and I happen to draw Gates via a random process. When I calculate the average net worth of my sample, it will absolutely be skewed towards the highest ends of net worth. Currently Bill Gates is worth $132.6B. Consider this: even if the rest of the 29 instructors have $0 net worth, by having Gates in there would still make for an average net worth of my campus instructors would be $4.42B!! That a B for Billionaire! Cleary that does not represent the true net worth of most instructors. If I want to have a real beat on how much instructors are worth, I will have to kick Bill out.

Similar story for dropping the 150% increase outlier for 100mg. It is such a massive difference from what “usually” happens, we should rightly consider that outlier to be at best an extreme case of AZ, a case that’s atypical.

According to the study details on ClinicalTrials.gov (Trial ID: NCT04079803) it was reported the data of patients who showed no simufilam in the blood after was dropped from analysis and that included a few patients in the 100mg group. It’s my speculation that the outlier was one of these 2 patients, but somehow one of the patients’ data remained in the data sets in error. This could help explain why Remi said “the plots are wrong, but the data analysis is correct:” It’s possible that the data analysis was run on the updated data, but the plots were generated by the original data set. Indeed, $SAVA’s presentation does state that the 100mg and 50mg groups showed the mean decline of PTau181 to be roughly 17% and 15% respectively, which is exactly what my analysis shows.

Why does having no simufilam matter? Again, I am not a doctor or any sort of medical professional, but I’m not completely ignorant on the subject either. Having no simufilam in your blood could indicate several things, but one possibility comes to mind: for some reason the patients just cannot process the drug at all, its down the hatch and out of your body in a similar way that your dog eating an aluminum can would like “re-release” the can pretty much untouched. Again: I am not a medical professional, but it would make sense to boot the data of a patient whose body cannot process the drug at all.

Conclusion and Take Aways

• Yes, there are errors in the plots from $SAVA’s AAIC presentation
• I reran analysis on a complete data set using the same data extraction methods used by Dr. Bik
• Yes, there isn’t much we can deduce numerically from the data in we leave the outlier in the 100mg group
• There is a clear trend for most patients taking 100mg of Simufilam
• It is reasonable to dismiss an outlier in an analysis that aims to show efficacy for the majority of patients
• It is reasonable to dismiss an outlier IF they cannot process the drug at all
• With no outlier, it is very hard to argue that Simufilam has no effect on PTau181 level
• Always keep in mind that a reduction in PTau181 does not imply there will be an improvement in the progression of the disease
• Phase 3 will be a much longer-term study, placebo-controlled, randomly assigned double blind study with a much larger sample size.
• I did forget to mention that in general larger sample sizes result in even more specific confidence intervals, so Phase 3 will be the ultimate arbiter of Simufilam’s fate, and Remi is on record for acknowledging that
• It’s my opinion that the errors on plots were done unintentionally incorrect.
• It’s my opinion that leaving out the outlier in this case is warranted and not unusual in data analysis in biotech research.

Reaching Out To $SAVA

I will be directly contacting $SAVA and request the raw data on this study. If $SAVA responds and shares their data with me, I will happily update this article and include the analysis of the true raw data.

Links To Data, R Script, and everything else:

Links to relevant information:

• Picture used in data extraction: https://1drv.ms/u/s!Ai4CyarRu4OSg54zfEKAnmmvFQe0WQ
• JSON File of my data extraction that can be loaded into WebPlotDigitizer: https://1drv.ms/u/s!Ai4CyarRu4OSg540ynLzn7QambO3jQ?e=HHYz89
• Data File with outlier in 100mg group: https://1drv.ms/x/s!Ai4CyarRu4OSg6hj7hk_FfKDmsf-SQ?e=nVjzVB
• Data File with outlier removed from data set: https://1drv.ms/u/s!Ai4CyarRu4OSg6hmBsV-tVZhdz0neQ?e=BY9Qja
• The R-Script used to analyze the data: https://1drv.ms/u/s!Ai4CyarRu4OSg6hi0tCT91DGGDLN9A?e=t0hy3f