def dataProvider_testAddProc(self): pAddProc1 = Proc() pAddProc2 = Proc() pAddProc3 = Proc() pAddProc4 = Proc(tag = 'new') pAddProc5 = Proc(tag = 'new') pAddProc6 = Proc(tag = 'new') aggr = Aggr(pAddProc1, pAddProc2, pAddProc3) yield aggr, pAddProc4, 'starts', None, True, utils.uid(pAddProc4.tag + '@' + aggr.id, 4) yield aggr, pAddProc5, 'ends', None, True, utils.uid(pAddProc5.tag + '@' + aggr.id, 4) yield aggr, pAddProc6, None, None, False, utils.uid(pAddProc6.tag + '@' + aggr.id, 4)
def testInit(self, args, kwargs, exception=None, msg=None):
self.maxDiff = None
if exception:
self.assertRaisesRegex(exception, msg, Aggr, *args, **kwargs)
else:
aggr = Aggr(*args, **kwargs)
self.assertEqual(aggr.id,
'aggr' if 'id' not in kwargs else kwargs['id'])
for i, p in enumerate(aggr._procs.values()):
self.assertEqual(
p.tag, kwargs['tag'] if 'tag' in kwargs else utils.uid(
args[i].tag + '@' + aggr.id, 4))
self.assertEqual(p.aggr, aggr.id)
self.assertIs(aggr._procs[p.id], p)
if ('depends' not in kwargs
or kwargs['depends']) and len(args) > 0:
self.assertListEqual(aggr.starts,
[list(aggr._procs.values())[0]])
self.assertListEqual(aggr.ends,
[list(aggr._procs.values())[-1]])
for i, proc in enumerate(aggr._procs.values()):
if i == 0: continue
self.assertIs(proc.depends[0],
list(aggr._procs.values())[i - 1])
else:
self.assertListEqual(aggr.starts, [])
self.assertListEqual(aggr.ends, [])
for i, proc in enumerate(aggr._procs.values()):
if i == 0: continue
self.assertListEqual(proc.depends, [])
def testSuffix(self):
p = proc('tag_unique')
config = {
key: val
for key, val in p.config.iteritems() if key not in [
'workdir', 'forks', 'cache', 'retcodes', 'echo', 'runner',
'exportdir', 'exporthow', 'exportow', 'errorhow', 'errorntry'
] or key.endswith('Runner')
}
config['id'] = p.id
config['tag'] = p.tag
if config.has_key('callback'):
config['callback'] = utils.funcsig(config['callback'])
# proc is not picklable
if config.has_key('depends'):
depends = config['depends']
pickable_depends = []
if isinstance(depends, proc):
depends = [depends]
elif isinstance(depends, aggr):
depends = depends.procs
for depend in depends:
pickable_depends.append(depend.id + '.' + depend.tag)
config['depends'] = pickable_depends
if config.has_key('input') and isinstance(config['input'], dict):
config['input'] = copy.deepcopy(config['input'])
for key, val in config['input'].iteritems():
config['input'][key] = utils.funcSig(val) if callable(
val) else val
signature = pickle.dumps(str(config))
self.assertEqual(p._suffix(), utils.uid(signature))
def test_suffix_input(tmpdir):
p82 = Proc()
p82.input = 'a'
p82.input = [1]
sigs = OBox()
sigs.argv0 = str(Path(sys.argv[0]).resolve())
sigs.id = 'p82'
sigs.tag = 'notag'
assert p82.config.input == {'a': [1]}
sigs.input = {'a': "[1]"}
assert p82.suffix == uid(sigs.to_json())
def test_suffix_compute(tmpdir):
p76 = Proc()
p81 = Proc()
p81.depends = p76
p81.input = 'a'
sigs = OBox()
sigs.argv0 = str(Path(sys.argv[0]).resolve())
sigs.id = 'p81'
sigs.tag = 'notag'
sigs.input = 'a'
sigs.depends = ['p76#' + p76.suffix]
assert p81.suffix == uid(sigs.to_json())
def testUid (self):
import random, string
def randomword(length):
return ''.join(random.choice(string.lowercase) for i in range(length))
uids = {}
for i in range (10000):
s = randomword (10)
uid = utils.uid (s)
uids[uid] = 1
self.assertEqual (len (uids.keys()), 10000)
def dataProvider_testSuffix(self):
pSuffix = Proc()
pSuffix.props['suffix'] = '23lhsaf'
yield pSuffix, '23lhsaf'
pSuffix1 = Proc()
pSuffix1.input = {'in': lambda ch: ch}
pSuffix1.depends = pSuffix
config = {key:val for key, val in pSuffix1.config.items() if key in [
'id', 'tag', 'input'
]}
config['argv0'] = path.realpath(sys.argv[0])
if isinstance(config['input'], dict):
config['input'] = pycopy.copy(config['input'])
for key, val in config['input'].items():
config['input'][key] = utils.funcsig(val) if callable(val) else val
config['depends'] = [pSuffix.name(True) + '#' + pSuffix._suffix()]
yield pSuffix1, utils.uid(json.dumps(config, sort_keys = True))
def test_uid_uniq(randstrs):
uids = [uid(rstr) for rstr in randstrs]
assert len(set(uids)) == len(uids)
def test_uid(string, length, alphabet, expect):
assert uid(string, length, alphabet) == expect