Multiprocessing in Python Data Function

[Uwe ...]

Hello,

trying to run a Python data function that uses multiprocessing results in the error "_pickle.PicklingError: Can't pickle : attribute lookup Worker on builtins failed". When running the same script outside of Spotfire with the local Python interpreter, everything works fine. Does anyone have any idea what could be the cause for the different behavior of the script

Thanks Uwe

import multiprocessing as mp

import time

import random as rd

 

class Task(object):

def __init__(self, task:int) -> None:

self.__task = task

 

def __call__(self, worker:int) -> float:

t = rd.random()

time.sleep(t)

return(t)

 

class Worker(mp.Process):

def __init__(self, tasks:mp.JoinableQueue, results:mp.Queue) -> None:

mp.Process.__init__(self)

self.__tasks = tasks

self.__results = results

 

def run(self) -> None:

number = int(self.name.split('-')[1])

while True:

task = self.__tasks.get()

if task is None:

self.__tasks.task_done()

break

answer = task(number)

self.__tasks.task_done()

self.__results.put(answer)

 

 

return

 

class Master(object):

def run(self, numTasks:int, numWorker:int) -> None:

tasks = mp.JoinableQueue()

for i in range(numTasks):

tasks.put(Task(i))

for i in range(numWorker):

tasks.put(None)

results = mp.Queue()

for i in range(numWorker):

c = Worker(tasks, results)

c.start()

tasks.join()

return(sum([results.get() for i in range(results.qsize())])/numTasks)

 

 

rd.seed(time.time())

m = Master().run(10, 3)

[Alain Martens]

Hi Uwe,

Often different versions of python will not work properly while unpickling.

Have you tried to run your code using the localPython interpreter that ships with Spotfire Analyst outside of Spotfire to exclude version conflicts (The location can be found by going to Tools, Python Tools, path to local interpreter)

Which version of Spotfire are you using And are you able to share a dxp with an example

Alain

[Elijah Manier]

I have a similar issue I ran my code using the intepreter included with spotfire, the code runs normally.

[Uwe ...]

Hi Alain,

 

by "outside Spotfire" I meant the Python console, which I can start under the menu item "Python Tools". The Spotfire version is 11.4.2. I cannot save a dxp with the embedded data function because when you insert a data function you have to run it once before you can save it.

 

Uwe.

[Alain Martens]

Hi Uwe,

Ok. Thanks for the info. It would be best if we're able to reproduce the behaviour you are seeing. It's indeed true that a data functionneeds to run first before it can be saved. To work around this, could you save a dummy script that can run once and then update the data function with your code

Thanks,

Alain