Custom error bar functions

[Melchior du Lac]

Hello community, I would like to add custom error bars in my line plot, but I'm struggling to find a way. More precisely, I would like to have error bars with confidence intervals that have different confidence levels. In the Error Bars settings dropdown there is both U95 and L95 for the lower end and upper end of the error bars, but only for the 80% confidence interval. 

I understand that you can create a "calculated column" and use that as the input, but I do not think that it will work for me. 

Is there a way to add custom equations to the "Upper Error" and "Lower Error"?

Thanks a lot 

[Gaia Paolini]

Can you add more details to your question? I don't see an option to add U95 or L95 errors to a line chart, only an option to select an expression to represent the Upper Error and Lower Error. You can add any custom expression you want.

[Melchior du Lac]

Strange. I have Spotfire version 10.3.3.11 by the way.

I would like to display error bars that represent confidence intervals that are not the traditional 95% confidence level. I would like to make one with 80%. To do that I need to use a custom expression with mean+0.8*(std/sqrt(len)). Not sure where to put that.

Attached is a screenshot of what I see:

 

How does one add custom expressions from there?

Thanks I appreciate it

[Gaia Paolini]

To reach the expression editor, you would right click on the drop-down list for the upper/lower error. Would it be possible to post an anonymized sample dxp with your current U95 calculation, so I understand what you are trying to do?

[Melchior du Lac]

Ok I've made some progress. I'll try to post an anonymized version of the DXP, but we could do it with any example DXP file to be honest. I'll try to find one online.

I actually found a way to input a custom expression by expanding the error bars option as shown in the following screenshot. I used https://docs.tibco.com/pub/sfire-analyst/latest/doc/html/en-US/TIB_sfire-analyst_UsersGuide/stat/stat_confidence_intervals.htm to input the expression results.

I also tried to input a custom R expression in "Data Function properties" -> "Expression Function". And then choosing the name of the custom expression in in the custom error bars function as above. Also shown in the screenshot:

However both seem to fail for some of the traces (see below). Where if I choose a custom default aggregation function in the dropdown of the error bars then I get all points to have error bars, but with custom functions only one trace has error bars.

[Gaia Paolini]

Sorry but I am struggling to understand what the use case is, there is a lot of info you provide but I am unable to work out what the structure of the plot, the axes, etc is.

[Melchior du Lac]

Sorry for the avalanche of information. Please find attached a DXP example file that contains the two different examples that I mentioned in the previous post (with two different CIs).

With the very simple dataset in the DXP I pasted everything works. Need to figure out why it doesn't work in my original dashboard. What I need to understand is how data is passed to each functions.... Any help would be appreciated

[Gaia Paolini]

The expression function takes in a vector for each argument. So your input1 will be the entire column you specified.

Every operation then needs to be vectorized.

I tried this and it seemed to yield the same result (for what you called U80/L80, although I got a it lost in the versions). 

Maybe a bit long and overkill but felt clearer. It relies on the data.table library being installed. Say it is called NEW_DF:

library(data.table) dt=data.table('value'=input1,'groupby'=input2) #group statistics by year or whatever input categorydt[,stdv:=sd(value),by = c("groupby")] dt[,N:=.N,by = c("groupby")]dt[,Q:=sapply(N,function(x) {qt(0.80,x-1)}),by = c("groupby")] doit =function(a,b,c) { a*b/sqrt©}dt[,U:=mapply(doit,Q,stdv,N)] output = dt$U

you would then use these custom expressions for the upper and lower error respectively

Avg([people]) + Avg(NEW_DF([people],[year]))

Avg([people]) - Avg(NEW_DF([people],[year]))

annoyingly, it wants some aggregation in front of the function call.

I think you might parametrize it to some extent by putting the 0.80 into a third argument, like this:

library(data.table) dt=data.table('value'=input1,'groupby'=input2) quantile = unique(input3) #group statistics by year or whatever input categorydt[,stdv:=sd(value),by = c("groupby")] dt[,N:=.N,by = c("groupby")]dt[,Q:=sapply(N,function(x) {qt(quantile,x-1)}),by = c("groupby")] doit =function(a,b,c) { a*b/sqrt©}dt[,U:=mapply(doit,Q,stdv,N)] output = dt$U

and then calling it like this:

Avg([people]) + Avg(NEW_DF([people],[year],0.8))

but I cannot guarantee it works as expected, did not have time to test it.

Hope it helps