Understanding living systems
This shows you the differences between two versions of the page.
i:placenta:data [2018/03/26 17:21] Igor Chernyavsky |
i:placenta:data [2022/03/07 13:23] (current) Igor Chernyavsky [Running COMSOL] |
||
---|---|---|---|
Line 8: | Line 8: | ||
Please note that all data created in ''/dls/tmp/'' will be **deleted after 30 days**, so make sure you transfer it to Dropbox or other online storage at least weekly. | Please note that all data created in ''/dls/tmp/'' will be **deleted after 30 days**, so make sure you transfer it to Dropbox or other online storage at least weekly. | ||
+ | ==== Registration & Data Management ==== | ||
+ | |||
+ | New users need these two steps before moving any further: | ||
+ | * [[https://uas.diamond.ac.uk/uas/?y=2#register | FedID registration ]] | ||
+ | * [[http://www.diamond.ac.uk/Users/UserGuide/Before-you-Arrive/Safety-Video-and-Test.html | Safety Training]] | ||
+ | |||
+ | **[[https://icat.diamond.ac.uk/#/login | DLS Data Archive ]]** (accessing long-term storage for scans older than 40 days; see [[https://www.diamond.ac.uk/Users/Experiment-at-Diamond/IT-User-Guide/Not-at-DLS/Retrieve-data.html|data retrieval]] for more tips). | ||
+ | |||
+ | [[https://savu.readthedocs.io/en/latest/user_guides/user_training/ | Savu ]] is a preferred Python-based tool for reconstructing raw DLS tomography data (i.e. radiographic projections). | ||
==== Setting-up and Connecting ==== | ==== Setting-up and Connecting ==== | ||
Line 20: | Line 29: | ||
From now on, all you need to do is to double-click on the desktop NoMachine link and enter your //Federal User ID// and password. Note that you will need to select (or create a new) //Virtual Desktop//. | From now on, all you need to do is to double-click on the desktop NoMachine link and enter your //Federal User ID// and password. Note that you will need to select (or create a new) //Virtual Desktop//. | ||
+ | Alternatively, you could connect via ''ssh'' (e.g. to run a download operation remotely: ''wget https...'') | ||
+ | $ YourFedID@nx.diamond.ac.uk | ||
==== Remote SuRVoS operation ==== | ==== Remote SuRVoS operation ==== | ||
Run the following commands in a Terminal: ''Applications --> System Tools --> Terminal''. | Run the following commands in a Terminal: ''Applications --> System Tools --> Terminal''. | ||
<code> | <code> | ||
+ | $ module load hamilton | ||
+ | $ qlogin -P i13 -l gpu=1 -l gpu_arch=Pascal -l exclusive | ||
$ module load survos | $ module load survos | ||
$ survos & | $ survos & | ||
Line 32: | Line 45: | ||
---- | ---- | ||
\\ | \\ | ||
- | Other technical info (could be obsolete) useful for troubleshooting: | + | Technical info useful for troubleshooting: |
- | Linux Workstation [i13-ws010.diamond.ac.uk] 172.23.113.76 (NX port 4000) | + | ''Linux Imaging Workstation [i13-ws010.diamond.ac.uk] 172.23.113.76 (NX port 4000)''\\ |
- | Compute Server [cs04r-sc-serv-95] 172.23.146.36 | + | List of Compute Nodes: ''qhost''\\ |
- | GPU / High Memory Usage | + | Other possible graphical options for high GPU / Memory Usage: |
<code> | <code> | ||
- | $ module load global/cluster | + | module load global/cluster |
- | $ qlogin -q high.q@@com14 -l exclusive -l gpu_arch=Pascal -P i13 | + | qlogin -q high.q@@com14 -l exclusive -l gpu=1,nvidia_tesla -P i13 |
</code> | </code> | ||
- | Other graphical options: ''qlogin -q high.q@@com14 -l exclusive -l gpu=1,nvidia_tesla -P i13'' | + | |
+ | Similarly, to run Avizo 2019.1 (//on DLS campus only//) | ||
+ | <code>module load avizo/2019.1; avizo</code> | ||
Further details are available in the [[http://www.diamond.ac.uk/Beamlines/Mx/I24/I24-Manual/Remote-Access/Connection-to-Diamond/data-processing.html|online beamline manual]]. | Further details are available in the [[http://www.diamond.ac.uk/Beamlines/Mx/I24/I24-Manual/Remote-Access/Connection-to-Diamond/data-processing.html|online beamline manual]]. | ||
Line 67: | Line 82: | ||
<div rightalign>--- //Igor Chernyavsky, 2018/03/26 18:00// ---</div> | <div rightalign>--- //Igor Chernyavsky, 2018/03/26 18:00// ---</div> | ||
+ | |||
+ | |||
+ | ===== Maths Compute Servers at Alan Turing Bldg ===== | ||
+ | |||
+ | You need a a Maths Linux account to access the servers. If you do not have one, please contact Chris Paul, stating your UoM username and reason for access. | ||
+ | |||
+ | * **On Linux or MacOS**: open a //Terminal emulator// and run\\ | ||
+ | ''$ ssh -Y **username**@e-a07maat1101<wrap em>X</wrap>.it.manchester.ac.uk'' | ||
+ | |||
+ | * **On Windows**: install and run [[https://www.putty.org|PuTTY]]. Enter ''e-a07maat1101**X**.it.manchester.ac.uk'' as the //Host Name//, SSH as the //Connection Type// and hit [Open]. | ||
+ | |||
+ | Here **username** is your UoM username, and ''<wrap em>X</wrap>'' is the reference letter ('a' to 'l') from the Table below (if unsure, use **''a''** for CS1 as a starting point). | ||
+ | |||
+ | **Note 1**: If you are using MacOS or Windows, you also need to install and run an ''X Server'' first (see more details on [[https://kb.iu.edu/d/bdnt|X-forwarding]]). | ||
+ | |||
+ | **Note 2**: On a university-managed Linux PC, you could connect directly via a name alias, e.g. ''$ ssh -Y cs1'' .\\ \\ | ||
+ | |||
+ | |||
+ | ^ Ref (X) ^ Name ^ Core Count ^ Core Speed and Type ^ RAM (GiB) ^ Note ^ | ||
+ | | a | cs1 | 12 | 3.4 GHz (Intel Xeon E5-2643v3) | 768 | Memory-intensive | | ||
+ | | b | cs2 | 8 | 3.3 GHz (Intel Xeon E5-2643) | 128 | | | ||
+ | | c | cs3 | 8 | 3.3 GHz (Intel Xeon E5-2643) | 128 | | | ||
+ | | d | cs4 | 8 | 3.3 GHz (Intel Xeon E5-2643) | 128 | | | ||
+ | | e | cs5 | 12 | 2.5 GHz (Inter Xeon E5-2430v2) | 128 | [offline] | | ||
+ | | f | cs6 | 12 | 2.5 GHz (Intel Xeon E5-2430v2) | 128 | | | ||
+ | | g | cs7 | 12 | 2.5 GHz (Intel Xeon E5-2430v2) | 128 | | | ||
+ | | h | cs8 | 12 | 2.5 GHz (Intel Xeon E5-2430v2) | 128 | | | ||
+ | | i | cs9 | 8 (x2) | 3.0 GHz (Intel Xeon E5-2623v3) | 256 | | | ||
+ | | j | cs10 | 8 (x2) | 3.0 GHz (Intel Xeon E5-2623v3) | 256 | no COMSOL | | ||
+ | | k | cs11 | 12 (x2) | 3.4 GHz (Intel Xeon 6128) | 1280 | CPU- & Memory-intensive; no COMSOL | | ||
+ | | l | cs12 | 12 | 2.5 GHz (Intel Xeon E5-2430v2) | 192 | | | ||
+ | | m | cs13 | 56 (x2) | 2.2 GHz (Intel Xeon 6238R) | 1024 | CPU- & Memory-intensive; 892GB SSD (/tmp) | | ||
+ | | n | cs14 | 56 (x2) | 2.2 GHz (Intel Xeon 6238R) | 1024 | CPU- & Memory-intensive; 892GB SSD (/tmp) | | ||
+ | | | minerva | 20 (x2) | 2.2 GHz (Intel Xeon 4114) | 1536 | Memory- & GPU-intensive (2x Nvidia P100 16GB); 2 TB HDD | | ||
+ | | | citadel | 8 (x2) | 3.4 GHz (Intel Xeon E5-1680v4) | 256 | Visualisation & GPU-intensive (Nvidia GTX1080 8GB); 8 TB HDD | | ||
+ | |||
+ | Note that cs1-cs8 cores run in a single-thread mode (HT is switched off). | ||
+ | |||
+ | |||
+ | System info: | ||
+ | <code> | ||
+ | free -h # RAM memory (or, $sudo dmidecode -t memory) | ||
+ | lscpu # CPU params | ||
+ | glxinfo -B # GPU memory | ||
+ | # further detailed info | ||
+ | sudo lshw -short #(omit sudo for partial info) | ||
+ | </code> | ||
+ | Load info: | ||
+ | <code> | ||
+ | top (followed by pressing the [t], [1] and [m] keys) | ||
+ | </code> | ||
+ | |||
+ | ==== Running COMSOL ==== | ||
+ | |||
+ | <code> | ||
+ | $ module load COMSOL/5.6 #or COMSOL/6.0 | ||
+ | $ comsol & | ||
+ | </code> | ||
+ | |||
+ | **Note 1**: If there are errors related to OpenGL, try | ||
+ | <code> | ||
+ | $ comsol -3drend sw & | ||
+ | </code> | ||
+ | |||
+ | **Note 2**: COMSOL is //not// available on compute servers ''cs10'' and ''cs11''. | ||
+ | |||
+ | **Note 3**: You could check available software versions by | ||
+ | <code> | ||
+ | $ module avail | ||
+ | </code> | ||
+ | |||
+ | To install COMSOL on a self-managed PC or laptop, download the [[https://livemanchesterac-my.sharepoint.com/:u:/g/personal/chris_paul_manchester_ac_uk/EUu_mH4g5TxOj2Zj02jPpdYBN5fD0RfE5BK9lPLB0RzF7Q?e=4%3aD1k4ox&at=9 | distributive [6 GB]]] (multi-platform ISO disk image, supporting Linux, MacOS and Windows) and use the following details during the setup: | ||
+ | <code>licence port@hostname: 15700@lfarm4.eps.manchester.ac.uk; licence number: 7076735</code> | ||
+ | |||
+ | ==== Running MATLAB ==== | ||
+ | |||
+ | <code> | ||
+ | $ module load matlab2017a | ||
+ | $ matlab & | ||
+ | </code> | ||
+ | |||
+ | |||
+ | ==== Other software ==== | ||
+ | |||
+ | [[https://manchester.saasiteu.com/Modules/SelfService/#knowledgeBase/view/AFC87036FF584A79ABFABA678D76FBA7 | ||
+ | | GraphPad Prism 8.0 Installer]] - see instructions. | ||
+ | |||
+ | [[http://www.itservices.manchester.ac.uk/software/ | UoM Research Software Repository]] | ||
+ | |||
+ | **Note**: When running your code at the University compute cluster (known as Computational Shared Facility, [[http://ri.itservices.manchester.ac.uk/csf3 | CSF]]), use the following to enable Internet access (e.g. to install necessary packages for Julia, Python, R, etc.): | ||
+ | <code> | ||
+ | module load tools/env/proxy | ||
+ | </code> | ||
+ | |||
+ | <div rightalign>--- //Igor Chernyavsky, 2021/07/22 15:00// ---</div> | ||
+ | |||
+ | |||
+ | |||
+ | ===== Mounting misc Remote file systems on Linux ===== | ||
+ | |||
+ | Before you start, make sure there is an empty directory (e.g. ''~/Shared'') in your ''home'' directory that you are going to mount. | ||
+ | |||
+ | * Mount UoM **RDS-SSH Data Share** | ||
+ | <code> | ||
+ | sshfs UoM_USERNAME@rds-ssh.itservices.manchester.ac.uk:/mnt/eps01-rds/Placental-Biophysics-Group/ ~/Shared/RDS/ | ||
+ | fusermount -u ~/Shared/RDS/ | ||
+ | </code> | ||
+ | |||
+ | * Mount UoM **P-Drive** | ||
+ | <code> | ||
+ | sudo mount -t cifs -o user=UoM_USERNAME,domain=ds.man.ac.uk,sec=ntlmsspi,uid=`id -u`,gid=`id -g` //nask.man.ac.uk/home$ ~/Shared/PDrive/ | ||
+ | sudo umount ~/Shared/PDrive/ | ||
+ | </code> | ||
+ | |||
+ | * Mount **Google Drive** via [[https://github.com/astrada/google-drive-ocamlfuse/|gdfuse]] | ||
+ | <code> | ||
+ | google-drive-ocamlfuse ~/Shared/GDrive/ | ||
+ | fusermount -u ~/Shared/GDrive/ | ||
+ | </code> | ||
+ | |||
+ | * Mount **Dropbox** via [[https://github.com/rianhunter/dbxfs|dbxfs]] (N.B. use '''-o nonempty''' option if sure; you might also need to install the following ''Ubuntu'' packages: ''libfuse2 build-essential libssl-dev libffi-dev python3-pip'') | ||
+ | <code> | ||
+ | dbxfs ~/Shared/Dropbox/ | ||
+ | fusermount -u ~/Shared/Dropbox/ | ||
+ | </code> | ||
+ | For uploading a large file (>~ 10GB) or multiple files, use [[https://github.com/andreafabrizi/Dropbox-Uploader|dropbox_uploader]] script: | ||
+ | <code> | ||
+ | ./dropbox_uploader -s -p upload /LOCAL_FOLDER /REMOTE_FOLDER | ||
+ | </code> | ||
+ | |||
+ | * Mount **DLS I13 Data Storage** (only available for **60 days** after the beamtime) | ||
+ | <code> | ||
+ | sshfs FedID_USERNAME@nx.diamond.ac.uk:/dls/i13/data/ ~/Shared/DLS/ | ||
+ | fusermount -u ~/Shared/DLS/ | ||
+ | </code> | ||
+ | |||
+ | <div rightalign>--- //Igor Chernyavsky, 2019/05/24 21:12// ---</div> | ||