How to copy marked visible rows to a new table?

[Henry Heberle]

The code below almost do the job, but it is selecting ALL marked items instead of bringing only VISIBLE marked items.

If the user selects 10 rows using a marker, then they filter, displaying only 5 of the marked rows in the visualisation/tabular view, then they click a button to run the script, the script is running on 10 rows instead of the selected & filtered 5 rows. This happens because filter and markers are independent. But how to query implementing the dependency in IronPython?
 

from Spotfire.Dxp.Data import DataFlowBuilder
from Spotfire.Dxp.Data.Import import DataTableDataSource
from Spotfire.Dxp.Data import RowSelection, IndexSet
 
# Source
tableName = "my_initial_table_to_select_from"
dt = Document.Data.Tables[tableName]
markings = Document.Data.Markings["My Marker"]

# filter colums 
ds = DataTableDataSource(Document.Data.Tables[tableName], markings)
dfb = DataFlowBuilder(ds, Application.ImportContext)
context = Document
flow = dfb.Build()

# All collumns are imported, and only marked rows:
try: # if the table already exists
	filtered_table = Document.Data.Tables["final_table_to_add_only_selected_visible_rows"]
	filtered_table.ReplaceData(ds)
except: # if the table was not created before
	Document.Data.Tables.Add("final_table_to_add_only_selected_visible_rows", flow)

 

Edited July 15 by Henry Heberle

[David Boot-Olazabal]

Hi Henry,

I've made a slight adjustment to your script, combining marking and filtering (for this part, you can also look here).
The script now looks like this:
 

from Spotfire.Dxp.Data import DataFlowBuilder
from Spotfire.Dxp.Data.Import import DataTableDataSource
from Spotfire.Dxp.Data import RowSelection, IndexSet
 
# Source
tableName = "SalesAndMarketing"
dt = Document.Data.Tables[tableName]
dataSelection=Document.ActiveFilteringSelectionReference
markings = Document.Data.Markings["Marking"]

# filter colums 
ds = DataTableDataSource(Document.Data.Tables[tableName], markings)
dfb = DataFlowBuilder(ds, Application.ImportContext)
context = Document
flow = dfb.Build()

# All collumns are imported, and only marked rows:
try: # if the table already exists
	filtered_table = Document.Data.Tables["final_table"]
	filtered_table.ReplaceData(ds)
except: # if the table was not created before
	Document.Data.Tables.Add("final_table", flow)

What is important though, is the order of things. Users need to filter first, before marking the data. Otherwise, only marked rows will be taken into account. But then the result set will be added to a new data table.
I have added an example dxp, but it's in Spotfire 14.4, so not sure if you would be able to open it (depends on your version of Spotfire).

Kind regards,

David

AddMarkedRowsToTable.dxp

[Henry Heberle]

Quote

 Users need to filter first, before marking the data.

This is exactly our problem. We can't control the users to always follow this order, or of making a mistake of thinking that they did in this order when they haven't.

We unfortunately have no pop-up or other visual indication warning the users that there are "marked rows" hidden by the filters, and users don't remember that they need to filter first. I'm also always using Ctrl+A and got myself in the situation of ending up selecting 100 IDs instead of 5 ids, which in turn resulted in many minutes loading data from the cloud (each ID brings lots of data).

In other context, marking many rows and then using filters to move between IDs or so, is very handy... when using "limit data by marking" in some visualisation. So the combination is powerful. But we have this problem that their independence can also be a problem, not only a solution.

We have the concept of selecting data using Markers in many parts of our dashboard, including in selection of input for AI Models and Stats, and download on demand data. So it's really important to be "user-proof" and really select or input into a model only "marked visible rows".

Here is the code for when our challenge was just selecting one ID. But now we need to copy rows from a table with 90 columns... and the unique identifier is the join of 15 columns... hence here we need Row index or so from Spotfire to be able to select the data correctly, the visible data.

 

from Spotfire.Dxp.Data import DataValueCursor, IndexSet

def get_visible_marked_rows(dataTable, markingName, columnName):
    # Create a cursor to access the column values
    values = DataValueCursor.CreateFormatted(dataTable.Columns[columnName])
    
    # Get the marked rows
    marking = Document.Data.Markings[markingName]
    markedRows = marking.GetSelection(dataTable).AsIndexSet()
    
    # Get the filtered rows
    filteredRows = Document.ActiveFilteringSelectionReference.GetSelection(dataTable).AsIndexSet()
    
    # Use the And method to find the intersection of marked and filtered rows
    visibleMarkedRows = IndexSet.And(markedRows, filteredRows)
    
    # Collect the values of the visible marked rows
    returnList = []
    for row in dataTable.GetRows(visibleMarkedRows, values):
        returnList.append(values.CurrentValue)
    
    return returnList

This won't work because it gets the "processed" data from UI it seems. E.g., numbers could even be converted to use , or . depending on the country of the user, etc.
It would also return all columns as text, and it would be very expensive computationally.

But the solution that we need would follow the same idea of the code above, but returning the marked visible rows and not the values of a column.

Spotfire 11
 

Edited July 15 by Henry Heberle

[David Boot-Olazabal]

Hi Henry,

Maybe this article gets you in the right direction.
It combines filtering and marking, and combines this. You would need to adapt this to your situation of course, but it could set you on the right path.

 

Kind regards,

David