Crawler

class managers.operatorsFiling.crawler.Crawler

Operator working under SimManager.

Available methods:

Method name	Method type
list_dirs()	class method
search_and_find()	static method
path_to_dir()	class method
path_to_file()	class method
show_files_with_path()	static method
show_files()	class method
smoke_out()	static method

NOTE:

• list_dirs returns list of of available same-level directories
• search_and_find returns full path to the directory/subdirectory/file
• path_to_dir returns path to a desired directory or subdirectory
• path_to_file returns path to a desired file in given directory path
• show_files_with_path returns dictionary as {"file1.ext": "its/path/file1.ext", "file2.ext": "a/path/file2.ext"}
• show_files returns dictionary of filenames and its path
• smoke_out return full path (os specific) to the directory/subdirectory/file

classmethod list_dirs(search_path=None)

Returns all same-level directores.

Keyword argument: search_path key whose value is a string of the form "path/to/search/for/subdir/in/a/dir".

Returned value: list of available directory names. An empty list is returned if there are no directories within the search_path.

Raised Exception: ValueError is returned if the search_path does not exist.

Use case:

• To list the available model scales list_dirs(search_path=os.getcwd()+os.sep+"models")

classmethod path_to_dir(dir_names=None)

Returns path to a desired directory or subdirectory.

Keyword argument:

Key	Value
dir_name	string or list of strings; strings for just “a_directory_name”; list of strings for [“a_dir_name”, “its_subdir”]

Returned value: full path (os specific) to the directory/subdirectory

Note: If only one directory name is given it is assumed this is available in the current working path (where this method is called from). This also applies to the first directory in the list of directory names.

Use case:

• Path to a directory in current working path

>> path_to_dir(dir_names="a_dir_name")

• Path to a subdirectory within the directory in current working path

>> path_to_dir(dir_names=["a_dir_name", "its_subdir"])

classmethod path_to_file(dir_names=None, file_name=None)

Returns path to a desired file.

Keyword arguments:

Key	Value
dir_name	string or strings list, where file is thought to reside. strings for just “a_directory_name”; list of strings for [“a_dir_name”, “its_subdir”]
file_name	string; “filename.ext”

Returned value: full path (os specific) to the filename

Note: If only one directory name is given it is assumed this is available in the current working path (where this method is called from). This also applies to the first directory in the list of directory names.

Use case:

• Path to a file in current working path

>> path_to_file(dir_names="a_dir_name", file_name="filename.ext")

• Path to a file in subdirectory within the directory in current working path

>> path_to_file(dir_names=["a_dir_name", "its_subdir"], file_name="filename.ext")

static search_and_find(search_type=None, working_dir=None, desired_name=None)

Searches for and finds files or directories.

Keyword arguments:

Key	Value
search_type	valid strings; “files” or “directories”
working_dir	path string; “current/working/directory”
desired_name	string or list of strings; strings for “file.name” or “a_directory_name”; list of strings for [“a_dir_name”, “its_subdir”]

Returned value: full path (os specific) to the directory/subdirectory/file

Raised Exceptions:

• ValueError if search_type is anything else other than “files” or “directories”
• ValueError if directory or file does not exist.

Use case:

Assuming present_dirpath = "/path/to/current/working/directory" then,

• To search for a subdirectory

>> search_for = ["a_dir_name", "its_subdir"]

>>search_and_find(search_type="directories", working_dir=present_dirpath, desired_name=search_for)

• To search for a directory

>> search_for = "a_dir_name"

>> search_and_find(search_type="directories", working_dir=present_dirpath, desired_name=search_for)

• To search for a file

>> search_for = "filename.ext"

>> search_and_find(search_type="files", working_dir=present_dirpath, desired_name=search_for)

classmethod show_files(dir_names=None)

Returns all the available files and its respective path.

Keyword argument:

Key	Value
dir_name	string or strings list, where file is thought to reside. strings for just “a_directory_name”; list of strings for [“a_dir_name”, “its_subdir”]

Returned value: dictionary with keys representing filenames and its path in respective value.

Note:

• If NO directory name is given it will find files in all the directory and subdirectory.
• If only one directory name is given it is assumed this is available in the current working path (where this method is called from). This also applies to the first directory in the list of directory names.

Use case:

• For files in current working path

>> show_files()

• For files in a subdirectory within the directory in current working path

>> show_files(dir_names=["a_dir_name", "its_subdir"])

static show_files_with_path(path_to_current_working_directory)

Searches for all available files within scope.

Argument: string of the form “current/working/directory”

Returned value: dictionary of the form {“file1.ext”: “its/path/file1.ext”, “file2.ext”: “a/path/file2.ext”}

Raised Exceptions: If dictionary is empty, returns "There are no files in the current path."

Use case:

Assuming present_dirpath = "/path/to/current/working/directory" then

• To show all the files in current working directory

>> show_files_with_path(os.getcwd())

• To show all the files in a desired directory

>> dirpath = "path/to/desired/directory"

>> show_files_with_path(dirpath)

static smoke_out(search_type=None, working_dir=None, desired_name=None)

Searches for and finds files or directories.

Keyword arguments:

Key	Value
search_type	valid strings; “files” or “directories”
working_dir	path string; “current/working/directory”
desired_name	string or list of strings; strings for “file.name” or “a_directory_name”; list of strings for [“a_dir_name”, “its_subdir”]

Returned value: full path (os specific) to the directory/subdirectory/file

Raised Exceptions:

• ValueError if search_type is anything else other than “files” or “directories”
• ValueError if directory or file does not exist.

Use case:

Assuming present_dirpath = "/path/to/current/working/directory" then

• To search for a subdirectory

>> search_for = ["a_dir_name", "its_subdir"]

>> search_and_find(search_type="directories", working_dir=present_dirpath, desired_name=search_for)

• To search for a directory

>> search_for = "a_dir_name"

>> search_and_find(search_type="directories", working_dir=present_dirpath, desired_name=search_for)

• To search for a file

>> search_for = "filename.ext"

>> search_and_find(search_type="files", working_dir=present_dirpath, desired_name=search_for)