def testGetOpenFilesList(): fileName = "/tmp/somefile" numFiles, filesList = getOpenFilesList() assert numFiles == 0 assert filesList == '' # empty string result f = open(fileName, 'w') numFiles, filesList = getOpenFilesList(offset=0) assert numFiles == 1 assert filesList.startswith(fileName) f.close() numFiles, filesList = getOpenFilesList() assert numFiles == 0 os.remove(fileName)
def testMainLogFileOpeningDuringDaemonisation(): py.test.skip("Test skipped, messes up with input/output streams for " "other test, better to run it stand-alone: " "py.test fdtcplib/test/test_fdtd.py -s -k testMainLogFileOpeningDuringDaemonisation") """ As described in the problem #41:comment:9, the main log file remains open thrice (under different descriptor) after initial daemonisation. In order to include this particular test into whole test suite running (issue with file descriptors being redirected, etc in fdtd.daemonize() function), perhaps after running this test, the file descriptors can be put back from backup sys.__stdout__, etc? """ c = """ [general] logFile = /tmp/fdtd.log pidFile = /tmp/fdtd.pid """ testName = inspect.stack()[0][3] f = getTempFile(c) inputOption = "--config=%s" % f.name conf = ConfigFDTD(inputOption.split()) f.close() logFile = "%s-%s" % (conf.get("logFile"), testName) pidFile = conf.get("pidFile") if os.path.exists(logFile): print("test: %s: file '%s' exists, removing ..." % (testName, logFile)) os.remove(logFile) if os.path.exists(pidFile): print("test: %s: file '%s' exists, removing ..." % (testName, pidFile)) os.remove(pidFile) logger = Logger(name=testName, logFile=logFile, level=logging.DEBUG) logger.debug("Before daemonization ...") numFiles, filesStr = getOpenFilesList() logger.debug("Logging open files: %s items:\n%s" % (numFiles, filesStr)) try: daemonize(conf, logger) except SystemExit: # previous leading process continues here return # here continues the newly created background daemon f = open(pidFile, 'r') rc = f.read() f.close() rc = rc.strip() pid = int(rc) numFiles, filesStr = getOpenFilesList() logger.debug("Logging open files: %s items:\n%s" % (numFiles, filesStr)) logger.debug("Before finishing ... ") logger.close() # the log file may be open more times due to streams descriptor # duplication as done in fdtd.daemonization() but now, once is # closed, there should not any other outstanding open file assert numFiles == 0
def testFDTDServiceOpenFilesFullTransfer(): """ #41:comment:8 - Too many open files (fdtd side) SendingClient actually removed itself from the executors container once it finishes so subsequent CleanupProcessesAction doesn't know about this process, nor about its open separate log file, which doesn't get closed. Simulate a simple successful transfer, send all actions and check number of open files - does all as it happens in fdtd.service() """ hostName = os.uname()[1] testName = inspect.stack()[0][3] initStateNumOpenFilesTestStart, filesStr = getOpenFilesList() print("%s: test 0: open files: %s items:\n%s" % (testName, initStateNumOpenFilesTestStart, filesStr)) # there should not be any open files now assert initStateNumOpenFilesTestStart == 0 f = getTempFile(functionalFDTDConfiguration) inputOption = "--config=%s --port=10001" % f.name confServer = ConfigFDTD(inputOption.split()) confServer.sanitize() loggerServer = Logger(name=testName, logFile="/tmp/fdtdtest-%s-writer.log" % testName, level=logging.DEBUG) apMon = None fdtdServer = FDTD(confServer, apMon, loggerServer) inputOption = "--config=%s --port=10002" % f.name confReader = ConfigFDTD(inputOption.split()) confReader.sanitize() loggerReader = Logger(name=testName, logFile="/tmp/fdtdtest-%s-reader.log" % testName, level=logging.DEBUG) apMon = None fdtdReader = FDTD(confReader, apMon, loggerReader) # -2 open log files, additional -1 is the temp config file initStateNumOpenFiles, filesStr = getOpenFilesList() print("%s: test 1: open files: %s items:\n%s" % (testName, initStateNumOpenFiles, filesStr)) assert initStateNumOpenFilesTestStart == initStateNumOpenFiles - 2 - 1 testActionServer = TestAction(hostName, hostName) testActionServer.id = testActionServer.id + "-writer" r = fdtdServer.service.service(testActionServer) assert r.status == 0 options = dict(gridUserDest="someuserDest", clientIP=os.uname()[1], destFiles=["/dev/null"]) recvServerAction = ReceivingServerAction(testActionServer.id, options) r = fdtdServer.service.service(recvServerAction) print r.msg assert r.status == 0 serverFDTPort = r.serverPort # there should be only 1 extra opened file now - ReceivingServerAction # separate log numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 2: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) assert initStateNumOpenFiles == numOpenFilesNow - 1 testActionReader = TestAction(hostName, hostName) testActionReader.id = testActionReader.id + "-reader" r = fdtdReader.service.service(testActionReader) assert r.status == 0 files = [TransferFile("/etc/passwd", "/dev/null")] # list of TransferFile options = dict(port=serverFDTPort, hostDest=os.uname()[1], transferFiles=files, gridUserSrc="soemuserSrc") sndClientAction = SendingClientAction(testActionReader.id, options) r = fdtdReader.service.service(sndClientAction) assert r.status == 0 # there should be +2 extra - for separate both server and client numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 3: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) # 2 extra files - separate transfer log at both ends assert initStateNumOpenFiles == numOpenFilesNow - 2 # now the transfer is over, both server (writer) and sender (reader) # parties kept their separate log files open, CleanupProcessesAction # will close them print "going to clean up" cl = CleanupProcessesAction(testActionReader.id, waitTimeout=False) r = fdtdReader.service.service(cl) assert r.status == 0 # one shall be closed now numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 4: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) assert initStateNumOpenFiles == numOpenFilesNow - 1 cl = CleanupProcessesAction(testActionServer.id, waitTimeout=False) r = fdtdServer.service.service(cl) assert r.status == 0 # both separate log files should be closed now # problem #41:comment:8 was here - server behaved correctly, but # reader kept its separate log file open numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 5: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) assert initStateNumOpenFiles == numOpenFilesNow fdtdServer.shutdown() fdtdServer.pyroDaemon.closedown() loggerServer.close() fdtdReader.shutdown() fdtdReader.pyroDaemon.closedown() loggerReader.close() # after even log files were closed, etc numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 6: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) # -1: the temp configuration file is still open assert initStateNumOpenFilesTestStart == numOpenFilesNow - 1
def testMainLogFileOpeningDuringDaemonisation(): py.test.skip( "Test skipped, messes up with input/output streams for " "other test, better to run it stand-alone: " "py.test fdtcplib/test/test_fdtd.py -s -k testMainLogFileOpeningDuringDaemonisation" ) """ As described in the problem #41:comment:9, the main log file remains open thrice (under different descriptor) after initial daemonisation. In order to include this particular test into whole test suite running (issue with file descriptors being redirected, etc in fdtd.daemonize() function), perhaps after running this test, the file descriptors can be put back from backup sys.__stdout__, etc? """ c = """ [general] logFile = /tmp/fdtd.log pidFile = /tmp/fdtd.pid """ testName = inspect.stack()[0][3] f = getTempFile(c) inputOption = "--config=%s" % f.name conf = ConfigFDTD(inputOption.split()) f.close() logFile = "%s-%s" % (conf.get("logFile"), testName) pidFile = conf.get("pidFile") if os.path.exists(logFile): print("test: %s: file '%s' exists, removing ..." % (testName, logFile)) os.remove(logFile) if os.path.exists(pidFile): print("test: %s: file '%s' exists, removing ..." % (testName, pidFile)) os.remove(pidFile) logger = Logger(name=testName, logFile=logFile, level=logging.DEBUG) logger.debug("Before daemonization ...") numFiles, filesStr = getOpenFilesList() logger.debug("Logging open files: %s items:\n%s" % (numFiles, filesStr)) try: daemonize(conf, logger) except SystemExit: # previous leading process continues here return # here continues the newly created background daemon f = open(pidFile, "r") rc = f.read() f.close() rc = rc.strip() pid = int(rc) numFiles, filesStr = getOpenFilesList() logger.debug("Logging open files: %s items:\n%s" % (numFiles, filesStr)) logger.debug("Before finishing ... ") logger.close() # the log file may be open more times due to streams descriptor # duplication as done in fdtd.daemonization() but now, once is # closed, there should not any other outstanding open file assert numFiles == 0
def testFDTDServiceOpenFilesFullTransfer(): """ #41:comment:8 - Too many open files (fdtd side) SendingClient actually removed itself from the executors container once it finishes so subsequent CleanupProcessesAction doesn't know about this process, nor about its open separate log file, which doesn't get closed. Simulate a simple successful transfer, send all actions and check number of open files - does all as it happens in fdtd.service() """ hostName = os.uname()[1] testName = inspect.stack()[0][3] initStateNumOpenFilesTestStart, filesStr = getOpenFilesList() print("%s: test 0: open files: %s items:\n%s" % (testName, initStateNumOpenFilesTestStart, filesStr)) # there should not be any open files now assert initStateNumOpenFilesTestStart == 0 f = getTempFile(functionalFDTDConfiguration) inputOption = "--config=%s --port=10001" % f.name confServer = ConfigFDTD(inputOption.split()) confServer.sanitize() loggerServer = Logger(name=testName, logFile="/tmp/fdtdtest-%s-writer.log" % testName, level=logging.DEBUG) apMon = None fdtdServer = FDTD(confServer, apMon, loggerServer) inputOption = "--config=%s --port=10002" % f.name confReader = ConfigFDTD(inputOption.split()) confReader.sanitize() loggerReader = Logger(name=testName, logFile="/tmp/fdtdtest-%s-reader.log" % testName, level=logging.DEBUG) apMon = None fdtdReader = FDTD(confReader, apMon, loggerReader) # -2 open log files, additional -1 is the temp config file initStateNumOpenFiles, filesStr = getOpenFilesList() print("%s: test 1: open files: %s items:\n%s" % (testName, initStateNumOpenFiles, filesStr)) assert initStateNumOpenFilesTestStart == initStateNumOpenFiles - 2 - 1 testActionServer = TestAction(hostName, hostName) testActionServer.id = testActionServer.id + "-writer" r = fdtdServer.service.service(testActionServer) assert r.status == 0 options = dict(gridUserDest="someuserDest", clientIP=os.uname()[1], destFiles=["/dev/null"]) recvServerAction = ReceivingServerAction(testActionServer.id, options) r = fdtdServer.service.service(recvServerAction) print(r.msg) assert r.status == 0 serverFDTPort = r.serverPort # there should be only 1 extra opened file now - ReceivingServerAction # separate log numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 2: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) assert initStateNumOpenFiles == numOpenFilesNow - 1 testActionReader = TestAction(hostName, hostName) testActionReader.id = testActionReader.id + "-reader" r = fdtdReader.service.service(testActionReader) assert r.status == 0 files = [TransferFile("/etc/passwd", "/dev/null")] # list of TransferFile options = dict(port=serverFDTPort, hostDest=os.uname()[1], transferFiles=files, gridUserSrc="soemuserSrc") sndClientAction = SendingClientAction(testActionReader.id, options) r = fdtdReader.service.service(sndClientAction) assert r.status == 0 # there should be +2 extra - for separate both server and client numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 3: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) # 2 extra files - separate transfer log at both ends assert initStateNumOpenFiles == numOpenFilesNow - 2 # now the transfer is over, both server (writer) and sender (reader) # parties kept their separate log files open, CleanupProcessesAction # will close them print("going to clean up") cl = CleanupProcessesAction(testActionReader.id, waitTimeout=False) r = fdtdReader.service.service(cl) assert r.status == 0 # one shall be closed now numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 4: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) assert initStateNumOpenFiles == numOpenFilesNow - 1 cl = CleanupProcessesAction(testActionServer.id, waitTimeout=False) r = fdtdServer.service.service(cl) assert r.status == 0 # both separate log files should be closed now # problem #41:comment:8 was here - server behaved correctly, but # reader kept its separate log file open numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 5: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) assert initStateNumOpenFiles == numOpenFilesNow fdtdServer.shutdown() fdtdServer.pyroDaemon.closedown() loggerServer.close() fdtdReader.shutdown() fdtdReader.pyroDaemon.closedown() loggerReader.close() # after even log files were closed, etc numOpenFilesNow, filesStr = getOpenFilesList() print("%s: test 6: open files: %s items:\n%s" % (testName, numOpenFilesNow, filesStr)) # -1: the temp configuration file is still open assert initStateNumOpenFilesTestStart == numOpenFilesNow - 1