Jump to content

Multiprocessing in Python Data Function

Uwe ...

Recommended Posts


trying to run a Python data function that uses multiprocessing results in the error "_pickle.PicklingError: Can't pickle : attribute lookup Worker on builtins failed". When running the same script outside of Spotfire with the local Python interpreter, everything works fine. Does anyone have any idea what could be the cause for the different behavior of the script

Thanks Uwe

import multiprocessing as mp

import time

import random as rd


class Task(object):

def __init__(self, task:int) -> None:

self.__task = task


def __call__(self, worker:int) -> float:

t = rd.random()




class Worker(mp.Process):

def __init__(self, tasks:mp.JoinableQueue, results:mp.Queue) -> None:


self.__tasks = tasks

self.__results = results


def run(self) -> None:

number = int(self.name.split('-')[1])

while True:

task = self.__tasks.get()

if task is None:



answer = task(number)







class Master(object):

def run(self, numTasks:int, numWorker:int) -> None:

tasks = mp.JoinableQueue()

for i in range(numTasks):


for i in range(numWorker):


results = mp.Queue()

for i in range(numWorker):

c = Worker(tasks, results)



return(sum([results.get() for i in range(results.qsize())])/numTasks)




m = Master().run(10, 3)

Link to comment
Share on other sites

Hi Uwe,

Often different versions of python will not work properly while unpickling.

Have you tried to run your code using the localPython interpreter that ships with Spotfire Analyst outside of Spotfire to exclude version conflicts (The location can be found by going to Tools, Python Tools, path to local interpreter)

Which version of Spotfire are you using And are you able to share a dxp with an example


Link to comment
Share on other sites

  • 2 weeks later...

Hi Alain,


by "outside Spotfire" I meant the Python console, which I can start under the menu item "Python Tools". The Spotfire version is 11.4.2. I cannot save a dxp with the embedded data function because when you insert a data function you have to run it once before you can save it.



Link to comment
Share on other sites

Hi Uwe,

Ok. Thanks for the info. It would be best if we're able to reproduce the behaviour you are seeing. It's indeed true that a data functionneeds to run first before it can be saved. To work around this, could you save a dummy script that can run once and then update the data function with your code



Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...