text:
Proceedings of the 29th International Conference on Genome Informatics (GIW 2018): systems biology
Analysis of significant protein abundance from multiple reaction-monitoring data
Jongsu Jun1,
Jungsoo Gim2,
Yongkang Kim1,
Hyunsoo Kim3,4,
Su Jong Yu5,
Injun Yeo3,
Jiyoung Park3,
Jeong-Ju Yoo5,
Young Youn Cho5,
Dong Hyeon Lee5,
Eun Ju Cho5,
Jeong-Hoon Lee5,
Yoon Jun Kim5,
Seungyeoun Lee6,
Jung-Hwan Yoon5,
Youngsoo Kim3,4 &
Taesung Park1,7
Discovering reliable protein biomarkers is one of the most important issues in biomedical research. The ELISA is a traditional technique for accurate quantitation of well-known proteins. Recently, the multiple reaction-monitoring (MRM) mass spectrometry has been proposed for quantifying newly discovered protein and has become a popular alternative to ELISA. For the MRM data analysis, linear mixed modeling (LMM) has been used to analyze MRM data. MSstats is one of the most widely used tools for MRM data analysis that is based on the LMMs. However, LMMs often provide various significance results, depending on model specification. Sometimes it would be difficult to specify a correct LMM method for the analysis of MRM data. Here, we propose a new logistic regression-based method for Significance Analysis of Multiple Reaction Monitoring (LR-SAM).
Through simulation studies, we demonstrate that LMM methods may not preserve type I error, thus yielding high false- positive errors, depending on how random effects are specified. Our simulation study also shows that the LR-SAM approach performs similarly well as LMM approaches, in most cases. However, LR-SAM performs better than the LMMs, particularly when the effects sizes of peptides from the same protein are heterogeneous. Our proposed method was applied to MRM data for identification of proteins associated with clinical responses of treatment of 115 hepatocellular carcinoma (HCC) patients with the tyrosine kinase inhibitor sorafenib. Of 124 candidate proteins, LMM approaches provided 6 results varying in significance, while LR-SAM, by contrast, yielded 18 significant results that were quite reproducibly consistent.
As exemplified by an application to HCC data set, LR-SAM more effectively identified proteins associated with clinical responses of treatment than LMM did.
Discovering protein disease biomarkers is an urgently pressing issue in biomedical research [1]. Historically, the enzyme-linked immunosorbent assay (ELISA) is a highly accurate protein quantitation technique [2], representing the "gold standard" for measuring levels of specific proteins [3]. However, recent discoveries of many novel proteins, having no available antibodies, now limits the use of the ELISA method [4]. Moreover, for newly discovered proteins, development of high quality ELISA assays requires considerable time and resources [5]. Together, these shortcomings have created a need for a different technique of targeted protein quantitation [6]. One such technique, quantitative mass spectrometry of proteins has advanced significantly over the last decade, and several methods have been developed for relative quantification of targeted proteins, using this technique [7].
In recent years, multiple reaction monitoring (MRM) mass spectrometry has been developed as an attractive tool for targeted proteins and now represents a promising alternative to ELISA for quantification of proteins. To that end, MRM uses sequence-specific tandem mass spectrometer fragmentations of peptides that can provide highly selective measurements for peptide-containing proteins [8]. Thus, without enrichment or fractionation approaches, MRM assays allow for the quantitation of protein ions within ranges of low (μg/mL) to high (ng/mL) levels [5, 9]. Additionally, development time for MRM assays is relatively shorter and less expensive than that for ELISA, with no requirements (and thus no costs) for antibody development.
Due to these many advantages, MRM assays are being increasingly used in systems biology and clinical investigations [10,11,12,13]. There are some analysis tools to analyze MRM data. Skyline is representative tool to create and analyze MRM dataset [14]. ProteoSign gives some simple statistical analysis results to find differentially expressed proteins [15]. roHits-viz is a web-based tool for visualizing interaction of protein data [16]. However, the development of statistical methods to determine significantly abundant proteins by the MRM assay has not received enough attention, compared to improvement of the MRM assay technology itself [17]. For MRM data analysis, two-sample t-tests or paired t-tests have been applied to identify proteins that change in abundance between two groups [18,19,20]. To test for multiple groups, one-way analysis of variance (ANOVA) has been employed [21, 22].
Recently, a linear mixed model (LMM) approach was proposed for MRM data analysis, and was implemented in MSstats (v3.7.1) [17], resulting in its widespread adoption [23]. LMM approaches include a single parameter for a group effect, peptide effects, and group and peptide interaction effects, along with subject and run effects. LMMs include both subject and run effects that can be designated as either random or fixed, to identify significantly evident proteins by testing the group effect. Recently, we observed that the LMM approach often provides different p-values for the group effect, from the same data, depending on which effects are treated as random or fixed. In particularly, the LMM test results vary considerably when there is an interaction effect between group and peptide.
In this report, we propose a new logistic regression-based method for Significance Analysis of Multiple Reaction Monitoring (LR-SAM). Unlike LMMs, our LR-SAM approach uses a much smaller number of parameters. Moreover, our LR-SAM does not require inclusion of all the effects related to the run. Accordingly, our model does not need to specify run effects as random or fixed.
Since LR-SAM uses log2-transformed relative intensity values, it does not need to include run effects. We consider two separate cases when the effects of peptides in a protein are homogeneous or heterogeneous. For the significance test for proteins, we consider Wald type tests, likelihood ratio tests (LRTs), and score tests [24].
Through various simulation studies, we compared the performance of LMMs to our approach. We first observed cases in which an LMM approach did not preserve type I error, and we also examined cases in which LR-SAM methods performed better. Some tests, based on the LR-SAM method, were more powerful when the expression pattern of peptide between groups was heterogeneous.
To further establish the translational relevance of our method, we compared LMMs to our approaches in a clinical study to identify candidate serum biomarkers of sorafenib response and prognosis in patients with hepatocellular carcinoma (HCC). Sorafenib is a multikinase inhibitor mainly used for the treatment of various solid tumors including HCC [25]. However, the response rate of sorafenib is about 8% for HCC patients. Due to its high cost, it would be more economical and more beneficial to patients if Sorafenib is applied only to the patients with high chance of response. Moreover, there are no good markers to predict the patients' response to sorafenib [26]. From May 2013 to August 2014, 115 HCC patient serum samples were collected from Seoul National University Hospital as part of an ongoing HCC study. One hundred twenty-four candidate protein biomarkers, and hepatic disease-associated proteins, were chosen from 50,265 proteins, based on the LiverAtlas database [27]. Sorafenib responses were measured according to the modified response evaluation criteria in solid tumors (mRECIST) guidelines [28]. Patients with complete responses, partial responses, or stable disease were categorized as responders, while those with progressive disease were categorized non-responders. Both LMM approaches and LR-SAM methods were employed to analyze the MRM data.
MRM dataset structure
Table 1 shows the structure of MRM dataset. Sample ID indicates individual, Run the shared run membership of the endogenous, and Area Ratio the expression of peptide detected. The MRM data has a hierarchical structure such that one protein contains several peptides. Since our goal is to identify proteins that are significantly different between the group, we need to combine the summarized expression of peptides from the same protein efficiently. Also, the batch effect would occur when MRM experiments were performed in different batches separately. These characteristics of MRM make it difficult to use the standard analyses such as two sample t-test.
Table 1 MRM dataset structure
Review of LMM approach
For a given protein, suppose it is comprised of K peptides. Let yi, j(i), k, l denote the log2(intensity) value of the j-th subject, nested in the i-th group of the k-th peptide and the l-th run. Then, the LMM used in MSstats is given as follows:
$$ {\mathrm{y}}_{\mathrm{i},\mathrm{j}\left(\mathrm{i}\right),\mathrm{k},\mathrm{l}}=\mu +{G}_i+S{(G)}_{j(i)}+{P}_k+{R}_l+{\left(\mathrm{G}\times \mathrm{P}\right)}_{\mathrm{i},\mathrm{k}}+{\left(P\times R\right)}_{k,l}+{\epsilon}_{i,j(i),k,l}, $$
where μ is the global mean; Gi is the i-th group effect; S(G)j(i) representing the j-th subject effect nested in the i-th group; Pk stands for the k-th peptide effect; Rl stands for the l-th run effect, (G × P)ik stands for the interaction effect between the i-th group and the k-the peptide; and (P × R)kl signifies the interaction effect between the k-th peptide and the l-th run. When all the effects are treated as fixed, these parameters have the following restrictions: \( \sum \limits_{i=0}^2{G}_i=0 \), \( \sum \limits_{j(i)=1}^{J(i)}S{(G)}_{j(i)}=0 \), \( \sum \limits_{k=1}^K{P}_k=0 \), \( \sum \limits_{l=1}^L{R}_l=0 \), \( \sum \limits_{i=0}^2{\left(G\times P\right)}_{i,k}=0 \), \( \sum \limits_{k=1}^K{\left(G\times P\right)}_{i,k}=0 \), \( \sum \limits_{k=1}^K{\left(P\times R\right)}_{k,l}=0,\sum \limits_{l=1}^L{\left(P\times R\right)}_{k,l}=0 \), and \( {\epsilon}_{i,j(i),k,l}\sim N\left(0,{\sigma}_{\epsilon}^2\right) \). Here, G0 stands for the effect of a reference group of MRM data. When the subject and run effects are treated as random, the restrictions on S(G)j(i), Rl, and (P × R)kl are replaced by \( S{(G)}_{j(i)}\sim N\left(0,{\sigma}_S^2\right) \), \( {R}_l\sim N\left(0,{\sigma}_R^2\right) \) and \( {\left(P\times R\right)}_{kl}\sim N\left(0,{\sigma}_{P\times R}^2\right) \), respectively.
For most MRM data analyses, the investigator's interest lies in determining which proteins differ in abundance between two groups. Thus, the hypothesis of interest is given below for comparing two groups:
$$ {H}_0:{G}_1={G}_2,{\mathrm{H}}_1:{\mathrm{G}}_1\ne {\mathrm{G}}_2 $$
The MSstats uses the t-test for this hypothesis.
LR-SAM approach
Our proposed LR-SAM approach uses a log2-transformed relative intensity value instead of the original log2-transformed intensity value itself. A log2-transformed relative intensity value was derived as yi, j(i), k, l − y0, 0(0), k, l. In that scenario, Pk, Rl and (P × R)kl effects that share the same k and l values are removed. If we denote yj, k as a log2-transformed relative intensity value of the j-th subject and the k-th peptide, where j = 1,J(1),J(1) + 1,…,J(1) + J(2). Then, the log2-transformed relative intensity value yj, k stands for the y1, j(1), k, l − y0, 0(0), k, l for j = 1,…,J(1), and it stands for the y2, j(2), k, l − y0, 0(0), k, l for j = J(1) + 1,…,J(1) + J(2).
LR-SAM with fixed effect
Consider a logistic regression model using log2-transformed relative intensity value, as follows:
$$ logit\left(P\left({Z}_j=1\right)\right)=\alpha +{\beta}_1{y}_{j,1}+\dots +{\beta}_K{y}_{j,K}, $$
where Zj is a group indicator of the j-th subject that is assumed to follow a Bernoulli distribution; α is an intercept; and βk is the coefficient of the k-th peptide. Note that the βk values are related to G1 − G2 + (G × P)1, k − (G × P)2, k of model (1). Therefore, if we treat all βk's as fixed, the hypothesis for comparing the two groups is:
$$ {H}_0:{\beta}_1=\cdots ={\beta}_K=0 $$
To test (4), we consider the likelihood ratio test (L) with K degrees of freedom, as follows:
$$ \mathrm{L}=-2\left({l}_0\left({\widehat{\alpha}}_0\right)-l\left(\widehat{\alpha},{\widehat{\beta}}_1,\dots, {\widehat{\beta}}_K\right)\right) $$
Here, \( {\widehat{\alpha}}_0 \) is the maximum likelihood estimate (MLE) of α under the null hypothesis (see above). \( {l}_0\left({\widehat{\alpha}}_0\right) \) is the maximum likelihood value under the null hypothesis, and \( \widehat{\alpha},{\widehat{\beta}}_1,\dots, {\widehat{\beta}}_K \) are the MLEs of α, β1, …, βK, respectively. It is also known that L asymptotically follows a chi-square distribution, with K degrees of freedom.
A Wald type test (W) statistic for analysis, is given below:
$$ W={\left(\begin{array}{c}{\widehat{\beta}}_1\\ {}\vdots \\ {}{\widehat{\beta}}_K\end{array}\right)}^{\prime } Var{\left({\widehat{\beta}}_1,\dots, {\widehat{\beta}}_K\right)}^{-1}\left(\begin{array}{c}{\widehat{\beta}}_1\\ {}\vdots \\ {}{\widehat{\beta}}_K\end{array}\right) $$
Here, W also asymptotically follows a chi-square distribution, with K degrees of freedom under the null hypothesis. If we assume that the βk values are homogeneous, a Wald test with 1 degree of freedom (W1) can be considered as follows:
$$ {W}_1={\widehat{\beta}}_p{A}^{-1}{\widehat{\beta}}_p, $$
in which \( {\widehat{\beta}}_p \) is the weighted average of \( {\widehat{\beta}}_k \) as \( \sum \limits_{k=1}^K{t}_k{\widehat{\beta}}_k \), where tk is a weight given as \( \frac{1/ Var\left({\widehat{\beta}}_k\right)}{\sum_{k=1}^K1/ Var\left({\widehat{\beta}}_k\right)} \), and A is the variance of \( {\widehat{\beta}}_p \), given as \( {\left(\begin{array}{c}{t}_1\\ {}\vdots \\ {}{t}_K\end{array}\right)}^{\prime } Var\left({\widehat{\beta}}_1,\dots, {\widehat{\beta}}_K\right)\left(\begin{array}{c}{t}_1\\ {}\vdots \\ {}{t}_K\end{array}\right) \). Thus, W1 asymptotically follows a chi-square distribution, with 1 degree of freedom, under the null hypothesis. Moreover, if the βkvalues are homogeneous, Eq. (3) (shown above) can be reduced to the following model:
$$ logit\left(P\left({Z}_j=1\right)\right)=\alpha +{\beta}^{\ast}\sum \limits_{k=1}^K{y}_{j,k} $$
The hypothesis of interest from this model (8) is H0 : β∗ = 0, and a Wald test (WS) is given by:
$$ {W}_S={\widehat{\beta}}^{\ast }\ Var{\left({\widehat{\beta}}^{\ast}\right)}^{-1}\ {\widehat{\beta}}^{\ast } $$
Here, \( {\widehat{\beta}}^{\ast } \) is the maximum likelihood estimate of β∗ from the model (8), and WS asymptotically follows a chi-square distribution, with 1 degree of freedom.
LR-SAM with random effects
When effects of peptides were heterogeneous, we could detect significant heterogenous effect by assuming random effects on coefficients βk. This assumption is commonly used in meta-analysis or rare variant analysis of genetic study [24, 29]. If we assume that βk follows a normal distribution, with mean 0 and variance wkτ (where wk is a known prior weight), then the logistic regression model (3) is expanded to a mixed effect model, and the hypothesis (4) is equivalent to:
$$ {H}_0:\tau =0,{H}_1:\uptau \ne 0 $$
Since the hypothesis H0 : τ = 0 is on the boundary of the parameter space, the variance-component score test can be considered. The score test statistic of the variance-component for (10) is:
$$ {S}_{VC}={\left(\boldsymbol{Z}-{\widehat{\boldsymbol{\mu}}}_0\right)}^{\prime}\boldsymbol{K}\left(\boldsymbol{Z}-{\widehat{\boldsymbol{\mu}}}_0\right) $$
Here, \( {\widehat{\boldsymbol{\mu}}}_0 \) is the estimated probability under H0; K = YWY′, where Y = [Y1, …, YK] and Yk = (y1k, …, ynk)′; Z means the group indicator vector, and W is a diagonal matrix with the k-th element as wk.
It is known that SVC follows a mixture of chi-square distributions \( {\sum}_{k=1}^K{\uplambda}_k{\chi_{1,k}}^2 \), where χ1, k2's are independent chi-square distributions with 1 degree of freedom, and λk is the k-th eigenvalue of P1/2KP1/2 [24]. Here, \( \boldsymbol{P}={\widehat{\boldsymbol{V}}}^{-\mathbf{1}}-{\widehat{\boldsymbol{V}}}^{-\mathbf{1}}\mathbf{1}\left({\mathbf{1}}^{\prime }{\widehat{\boldsymbol{V}}}^{-\mathbf{1}}\mathbf{1}\right){\mathbf{1}}^{\prime }{\widehat{\boldsymbol{V}}}^{-\mathbf{1}} \), where \( \widehat{\boldsymbol{V}} \) is a diagonal matrix with the k-th element as \( {\widehat{\mu}}_{0k}\left(1-{\widehat{\mu}}_{0k}\right) \). For simplicity, we assume a flat prior weight given by wk = 1 for k = 1, …, K.
Simulation design
We next performed simulation studies to investigate the power of the LMM and LR-SAM approaches for comparing two groups, and whether they preserve type I error. For analysis, there were four LMMs available, depending on how the random or fixed effects were specified: (i) LMM(FF), with fixed subject and run effects, (ii) LMM(FR), with fixed subject effect and random run effects, (iii) LMM(RF), with random subject and fixed run effects, and (iv) LMM(RR), with random subject and run effects. For each simulated dataset, the best LMM, LMM(best), was selected among four LMMs with the smallest Akaike information criterion (AIC) value [30]. Thus, there were five LR-SAM test statistics, L, W, W1, WS, and SVC, for comparison.
Simulation data was generated from the model, using 1000 repetitions. The global mean, μ, was arbitrarily set to 15, while the reference effects, G0, S(G)0, (0), and (G × P)0, k, for k = 1, …, K, were set to 0. The normal distribution, with mean 0 and variance 0.5, was set as the error distribution, and the number of peptides was set to 4. For the random subject effect, we generated S(G)j(i) from the identical normal distribution independently, with mean 0 and variance 0.25 for i = 1, 2. For the fixed subject effect, we set S(G)j(i) as follows:
$$ S{(G)}_{j(i)}=-{e}_S+\frac{2\left(j(i)-1\right)}{J(i)-1}{e}_S\ \mathrm{for}\kern0.5em j(i)=1,\dots, J(i)\kern0.75em \mathrm{and}\kern0.5em i=1,2,\mathrm{where}\kern0.5em {e}_S>0\ \kern0.5em \mathrm{and}\kern0.5em {e}_S=\sqrt{3\frac{J(i)-1}{J(i)+1}}{\sigma}_S,\kern0.5em \mathrm{with}\kern0.5em {\sigma}_S^2=0.25. $$
The peptide effect, Pk, was considered as a fixed effect, and was set as follows:
$$ {P}_k=-{e}_P+\frac{2\left(k-1\right)}{K-1}{e}_P\kern0.5em \mathrm{for}\kern0.5em k=1,\dots, K,\kern0.5em \mathrm{where}\kern0.5em {e}_P>0\kern0.5em \mathrm{and}\kern0.5em {e}_P=\sqrt{3\frac{K-1}{K+1}}{\sigma}_P\kern0.5em \mathrm{with}\kern0.5em {\sigma}_P^2=0.1 $$
For the random run effect, we generated Rl from the identical normal distribution, independently, with mean 0 and variance 0.25. For the fixed run effect, we set Rl as follows:
\( {R}_l=-{e}_R+\frac{2\left(m-1\right)}{M-1}{e}_R\kern0.5em \mathrm{for}\kern0.5em m=1,\dots, M,\kern0.5em \mathrm{where}\kern0.5em {e}_R>0\kern0.5em \mathrm{and}\kern0.5em {e}_R=\sqrt{3\frac{M-1}{M+1}}{\sigma}_R,\kern0.5em \mathrm{with}\kern0.5em {\sigma}_R^2=0.25 \)
We then considered the peptide×run interaction effect as either random or fixed, followed by the type of run effect. For the random peptide by run interaction effect, we generated (P × R)k, l from the identical normal distribution independently, with mean 0 and variance 0.1. For the fixed peptide×run interaction effect, we set (P × R)k, l as follows:
$$ {\left(P\times R\right)}_{k,l}=\left({e}_{PR}-\frac{2\left(k-1\right)}{K-1}{e}_{PR}\right){\left(-1\right)}^l\kern0.5em \mathrm{for}\kern0.5em l=1,\dots, L\kern0.5em \mathrm{and}\kern0.5em k=1,\dots, K,\kern0.5em \mathrm{where}\kern0.75em {e}_{PR}>0\kern0.5em \mathrm{and}\kern0.5em {e}_{PR}=\sqrt{3\frac{K-1}{K+1}}{\sigma}_{PR},\kern0.5em \mathrm{with}\kern0.5em {\sigma}_{PR}^2=0.1 $$
Provided those parameters, we further considered four group effect scenarios (GSs), as follows:
$$ \mathrm{GS}\ 0:\kern0.5em {G}_i=0\kern0.5em \mathrm{for}\ \mathrm{all}\ \mathrm{i}\ \mathrm{and}\ \mathrm{GS}\ 1:\kern0.5em {G}_2-{G}_1=1/3 $$
For detecting the group×peptide interaction effect, we set (G × P)i, k, as follows:
$$ {\left(G\times P\right)}_{i,k}=\left({e}_{GP}-\frac{2\left(k-1\right)}{K-1}{e}_{GP}\right){\left(-1\right)}^i\kern0.5em \mathrm{for}\kern0.5em i=1,2\kern0.5em \mathrm{and}\kern0.5em k=1,\dots, K,\kern0.5em \mathrm{where}\kern0.5em {e}_{GP}>0\kern0.5em \mathrm{and}\kern0.5em {e}_{GP}=\sqrt{3\frac{K-1}{K+1}}{\sigma}_{GP}. $$
Here, using σGP values determined from the squared average of the interaction effect, we considered three interaction effect scenarios (ISs) for σGP, as follows:
$$ \mathrm{IS}\ 1:\kern0.5em {\sigma}_{GP}^2=0.05\kern0.5em \mathrm{and}\kern0.5em \mathrm{IS}\ 2:\kern0.5em {\sigma}_{GP}^2=0.1 $$
To identify candidate serum hepatocellular carcinoma (HCC) biomarkers for prognosis and response to the tyrosine kinase inhibitor sorafenib, data from 115 HCC patients who had undergone continuous administration of sorafenib for more than 6 weeks, were collected from Seoul National University Hospital, as part of an ongoing study between May 2013 and August 2014. HCC was diagnosed by histological or radiological evaluation, with reference to the American Association for the Study of Liver Diseases (AASLD) [31] or the European Association for the Study of the Liver (EASL) [32] guidelines. All procedures/analyses were approved by the Seoul National University Hospital Institutional Review Board (IRB protocol No. 0506–150-005). Sorafenib response was evaluated using the modified response evaluation criteria in solid tumors (mRECIST) [28], using independent radiologic assessments. Patients with complete response, partial response, and stable disease were categorized as responders, while those with progressive disease were categorized as non-responders.
Toward these objectives, a total of 115 serum samples were randomly separated into two batches: the 1st batch consisted of 65 samples and the 2nd batch consisted of 50 samples. Of those, the total number of responders was 40, and the number of non-responders was 75. One hundred twenty-four candidate protein biomarkers, known hepatic disease-associated proteins, were chosen from 50,265 proteins, based on the LiverAtlas database [27]. One to seven peptides, comprised within each protein, were measured, and counted (Table 2).
Table 2 The number of proteins of each number of peptides
Since there were some cases in which LMM(FF), LMM(FR), and L did not preserve type I error in our simulation studies, we applied LMM(RF), LMM(RR), W, W1, WS and SVC to analyze the MRM data. Quantile normalization, provided within the MSstats package [20], was employed for preprocessing. To adjust the mean differences between the two batches, batch indicators were included in models (1), (3), and (8) for MRM data analysis.
Simulation results
The type I error and the empirical power were estimated as the proportion of p-values under 0.05, out of 1000 repetitions. The type I error rates of LMMs and LR-SAM are summarized in Table 3. The true type of subject effect more strongly affected the type I error rate of LMMs, as compared to the run effect. Analogously, when the true subject effect was fixed, the four LMMs controlled the type I errors. Among the five LR-SAM tests, L could not control the type I error rate when the sample size was 20, while the other four LR-SAM tests could. When the sample sizes were 50 and 100, all LR-SAM tests controlled type I error. When the true subject effect was random, LMM(FF) and LMM(FR) could not control type I error, whereas LMM(RF) and LMM(RR) could. L did not control the type I error rate when the sample sizes were 20 and 50, while the other four LR-SAM tests did. The AIC value of LMM(FF) tended to be the smallest among the four LMMs, under any conditions. Thus, LMM(FF) was most frequently selected as the best LMM.
Table 3 Estimated type I error of LMMs and LR-SAM methods
Some LR-SAM tests showed increased power as the interaction effects became large. When there was only an interaction effect, without group effects, the power of the LMMs W1, and WS did not increase, as shown in Table 4. As the sample size and the interaction effect became large, W and SVC showed increased power. Moreover, SVC provided higher power than W, when the sample size was 50, while W provided higher power than SVC, when the sample size was 100.
Table 4 Estimated power of LMMs and LR-SAM methods for GS 0. Bolded number indicates power of methods were more than 0.8
The size of the group effect showed large effects on all LMMs and LR-SAM methods. The estimated powers of the five LMMs, and the LR-SAM methods for GS 1, are shown in Table 5. The powers of L, W, W1, and SVC were all affected by the size of the interaction effect, when the group effect scenario was GS 1, while those of the five LMMs did not. As the interaction effect increased, the powers of L, W, and SVC increased, while the powers of four of the LMMs, and WS, did not, and the power of the W1 method even decreased. When the interaction scenario was IS 1, the powers of the L and SVC methods were higher than those of the LMMs, for a sample size of 100. For the IS 2 scenario, once the sample size exceeded 50, the L and SVC methods provided higher powers than those of the LMMs.
Collectively, as the size of group effects became large, all methods provided increased power. As the interaction effect became large, only the L, W, and SVC methods provided increased power.
HCC data analysis
Using the approaches described above, the maximum –log10-transformed p-values of the linear mixed models (LMMs) were higher than those of the LR-SAM methods. However, the correlation of the transformed p-values of LMM(RF) and LMM(RR) was lower than that of LR-SAM methods, as shown in Fig. 1. The correlation between LMM(RF) and LMM(RR) was 0.6558, while the minimum correlation among the LR-SAM methods was 0.784. The correlation between WS and SVC was the highest (0.9627), among the LR-SAM methods.
Pairwise scatter plot of –log10-transformed p-values from the LMM(RF), LMM(RR), W, W1, WS and SVC models based on multiple reaction monitoring (MRM) data. Vertical and horizontal dashed red lines represent Bonferroni-corrected significance levels, −log10(0.05/124). Diagonally dashed gray line represents one to one slope
The lowest p-values of LMM(RF) and LMM(RR) were lower than those of the LR-SAM methods, valued at 1.87 × 10−18 and 1.09 × 10−30, respectively. The lowest p-value of W, W1, and WS was 2 × 10−7, and the lowest p-value of SVC was 3.49 × 10−14 (Table 6). Although the lowest p-values of the LR-SAM methods were less significant than those of the LMMs, and for many proteins, LR-SAM methods provided higher significance results (Fig. 1).
Table 6 List of proteins and their p-values simultaneously identified by LMM or LR-SAM methods
The LMM(RF) and LMM(RR) methods identified 11 and 37 proteins, respectively. There were six (14.29%) proteins simultaneously identified by LMM(RF) and LMM(RR). For LR-SAM methods, W, W1, WS, and SVC identified 28, 19, 22, and 29 proteins, respectively. Among these proteins 18 (52.94%) were simultaneously identified by four methods. Finally, there were four (7.84%) proteins identified by all six methods of LMM and LR-SAM (Fig. 2).
Venn diagram of proteins identified from LMM (RF, RR) and LR-SAM (W, W1, WS and SVC), under a Bonferroni correction significance level of −log10(0.05/124)
All LMMs and LR-SAM methods simultaneously provided significance results for the presence of the proteins GPX3, IGHG1, IGHG3 and IGJ. Among these, IGJ (immunoglobulin J chain, linker protein for immunoglobulin alpha and mu) was previously reported as having a significant difference of expression in HCC tumors [33], while GPX3 (glutathione peroxidase 3) was reported as a tumor suppressor in HCC [34].
There were 14 proteins that the LR-SAM methods, but not the LMMs, simultaneously provided significance results of expression (Table 6). Among those 14, seven were previously reported, including FBLN1 (Fibulin-1), a tumor suppressor gene in HCC [35], SHBG (sex hormone-binding globulin), a prediagnostic risk marker for HCC [36], and LG3BP (galectin-3-binding protein), a potential marker in six cancer types, including HCC, lymphoma, NPC (nasopharyngeal carcinoma), CRC (colorectal carcinoma), and oral cancers [37]. Additionally, CATB (Cathepsin B) was previously reported as a potential candidate cancer biomarker in HCC [38], HPT (haptoglobin) was reported to associate with tumor progression in HCC [39], while POSTN (periostin) was reported as a marker for malignant transformation of hepatocytes [40]. Similarly, 14–3-3S (14–3-3 protein sigma) was reported as downregulated in HCC [41].
There were two proteins that LMMs simultaneously provided a significance result, while LR-SAM methods did not. Resultant p-values for the simultaneously identified 20 proteins are shown in Table 6. Two proteins that demonstrated significance by LMM(RF) and LMM(RR) were also previously reported: CD5L (CD5 antigen-like), reported as differentially expressed in hepatitis C patients [42]: and APOA4, (apolipoprotein A-IV), reported as misexpressed in liver metabolic disorders [43].
We examined the performance of LMMs and LR-SAM methods, through extensive simulation studies. When the true subject effect was random, LMM(FF) and LMM(FR) did not preserve type I error (Table 3). We also applied Akaike information criterion (AIC), for model selection, to check whether or not the best-performing LMM preserved type I error. However, our empirical study showed that LMM(FF) had the smallest AIC value for most simulation settings, which made it difficult to use AIC as a model selection criterion. On the other hand, LR-SAM methods, except L, well preserved type I error under any type of subject and run effects.
Since the hypothesis (2) did not consider the interaction effect, the power of the LMM approach was not affected under any size of the interaction effect. However, when the interaction effect (without the group effect) was considered, hypothesis (2) could not perform well, as we observed through our simulation studies. On the other hand, testing hypotheses (8) with W and (10) with SVC, changes under any size group effect and interaction effect were detectable. Additionally, LR-SAM methods did not provide higher power than LMMs, when there were no or only weak interaction effects.
Although LMM provided more protein significance results than LR-SAM methods, the proteins identified by the LMM(RF) and LMM(RR) approaches were very different from each other. On the other hand, LR-SAM methods provided more consistent significance results than those of the LMMs. Moreover, LR-SAM methods provided greater significance results than those identified by LMM(RF), for most proteins, and several proteins were identified only by LR-SAM methods. However, there was a still performance difference between the choices of fixed and random effect models in LR-SAM methods, as was in the LMM. This difference was caused by different testing hypotheses between W and SVC in LR-SAM. In addition, note that SVC allows the heterogenous effects of peptides, while W does not.
LMMs have been widely used to identify proteins with significantly altered abundance, in distinct disease states, based on MRM assays [39]. However, we found that LMM approaches provide inconsistent significance results for the same MRM data, depending on which effects are treated as random or fixed by simulation results. It is a well-known property of LMMs that the variance of model parameters are underestimated, when the fixed effect model is fitted but the true effects are random, and vice versa [44]. As a result, the protein significance results of LMMs may vary, depending on whether the true effect is random or fixed, and we also observed this phenomenon, as shown in Fig. 1 and Table 6. Thus, it is highly important to correctly specify the effect as random or fixed.
In this paper, we propose a new logistic regression-based method for Significance Analysis of Multiple Reaction Monitoring (LR-SAM). Unlike LMMs, our LR-SAM approach uses a much smaller number of parameters. Moreover, our LR-SAM does not require inclusion of all the effects related to the run. Accordingly, our model does not need to specify run effects as random or fixed.
In simulation study, our LR-SAM preserved type I error when the true subject effect was random, while LMM(FF) and LMM(FR) did not. In real data analysis, although LMM provided more protein significance results than LR-SAM methods, LR-SAM methods provided more consistent significance results than those of the LMMs. Thus, our proposed method, LR-SAM could give more reliable results than previous protein studies.
AASLD:
American Association for the Study of Liver Diseases
Akaike information criterion
EASL:
European Association for the Study of the Liver
ELISA:
GS:
Group effect scenarios
Interaction effect scenarios
LMM:
Linear mixed model
mRECIST:
Modified response evaluation criteria in solid tumors
MRM:
Multiple reaction monitoring
Frantzi M, Bhat A, Latosinska A. Clinical proteomic biomarkers: relevant issues on study design & technical considerations in biomarker development. Clin Transl Med. 2014;3(1):7.
Shi T, Su D, Liu T, Tang K, Camp DG, Qian WJ, Smith RD. Advancing the sensitivity of selected reaction monitoring-based targeted quantitative proteomics. Proteomics. 2012;12(8):1074–92.
Pan S, Aebersold R, Chen R, Rush J, Goodlett DR, McIntosh MW, Zhang J, Brentnall TA. Mass spectrometry based targeted protein quantification: methods and applications. J Proteome Res. 2008;8(2):787–97.
Shi T, Fillmore TL, Sun X, Zhao R, Schepmoes AA, Hossain M, Xie F, Wu S, Kim J-S, Jones N. Antibody-free, targeted mass-spectrometric approach for quantification of proteins at low picogram per milliliter levels in human plasma/serum. Proc Natl Acad Sci. 2012;109(38):15395–400.
Haab BB, Paulovich AG, Anderson NL, Clark AM, Downing GJ, Hermjakob H, LaBaer J, Uhlen M. A reagent resource to identify proteins and peptides of interest for the Cancer community a WORKSHOP REPORT. Mol Cell Proteomics. 2006;5(10):1996–2007.
Rifai N, Gillette MA, Carr SA. Protein biomarker discovery and validation: the long and uncertain path to clinical utility. Nat Biotechnol. 2006;24(8):971–83.
Mesri M. Advances in proteomic technologies and its contribution to the field of Cancer. Adv Med. 2014;2014:238045.
Liebler DC, Zimmerman LJ. Targeted quantitation of proteins by mass spectrometry. Biochemistry. 2013;52(22):3797–806.
Kuzyk MA, Smith D, Yang J, Cross TJ, Jackson AM, Hardie DB, Anderson NL, Borchers CH. Multiple reaction monitoring-based, multiplexed, absolute quantitation of 45 proteins in human plasma. Mol Cell Proteomics. 2009;8(8):1860–77.
Hale JE. Advantageous uses of mass spectrometry for the quantification of proteins. Int J Proteomics. 2013;2013:219452.
Carr SA, Abbatiello SE, Ackermann BL, Borchers C, Domon B, Deutsch EW, Grant RP, Hoofnagle AN, Hüttenhain R, Koomen JM. Targeted peptide measurements in biology and medicine: best practices for mass spectrometry-based assay development using a fit-for-purpose approach. Mol Cell Proteomics. 2014;13(3):907–17.
Picotti P, Aebersold R. Selected reaction monitoring-based proteomics: workflows, potential, pitfalls and future directions. Nat Methods. 2012;9(6):555–66.
Grebe SK, Singh RJ. LC-MS/MS in the clinical laboratory–where to from here? Clin Biochem Rev. 2011;32(1):5.
MacLean B, Tomazela DM, Shulman N, Chambers M, Finney GL, Frewen B, Kern R, Tabb DL, Liebler DC, MacCoss MJ. Skyline: an open source document editor for creating and analyzing targeted proteomics experiments. Bioinformatics. 2010;26(7):966–8.
Efstathiou G, Antonakis AN, Pavlopoulos GA, Theodosiou T, Divanach P, Trudgian DC, Thomas B, Papanikolaou N, Aivaliotis M, Acuto O, et al. ProteoSign: an end-user online differential proteomics statistical analysis platform. Nucleic Acids Res. 2017;45(W1):W300–6.
Knight JDR, Choi H, Gupta GD, Pelletier L, Raught B, Nesvizhskii AI, Gingras AC. ProHits-viz: a suite of web tools for visualizing interaction proteomics data. Nat Methods. 2017;14(7):645–6.
Chang C-Y, Picotti P, Hüttenhain R, Heinzelmann-Schwarz V, Jovanovic M, Aebersold R, Vitek O. Protein significance analysis in selected reaction monitoring (SRM) measurements. Mol Cell Proteomics. 2012;11(4):M111. 014662.
Huang SM, Bisogno T, Trevisani M, Al-Hayani A, De Petrocellis L, Fezza F, Tognetto M, Petros TJ, Krey JF, Chu CJ. An endogenous capsaicin-like substance with high potency at recombinant and native vanilloid VR1 receptors. Proc Natl Acad Sci. 2002;99(12):8400–5.
Zemann B, Kinzel B, Müller M, Reuschel R, Mechtcheriakova D, Urtz N, Bornancin F, Baumruker T, Billich A. Sphingosine kinase type 2 is essential for lymphopenia induced by the immunomodulatory drug FTY720. Blood. 2006;107(4):1454–8.
Mani D, Abbatiello SE, Carr SA. Statistical characterization of multiple-reaction monitoring mass spectrometry (MRM-MS) assays for quantitative proteomics. BMC Bioinformatics. 2012;13(Suppl 16):S9.
Yassine HN, Jackson AM, Reaven PD, Nedelkov D, Nelson RW, Lau SS, Borchers CH. The application of multiple reaction monitoring to assess ApoA-I methionine oxidations in diabetes and cardiovascular disease. Transl Proteomics. 2014;4:18–24.
Zhang P, Kirk JA, Ji W, dos Remedios CG, Kass DA, Van Eyk JE, Murphy AM. Multiple reaction monitoring to identify site-specific troponin I phosphorylated residues in the failing human heart. Circulation. 2012. https://doi.org/10.1161/CIRCULATIONAHA.112.096388.
Pursiheimo A, Vehmas AP, Afzal S, Suomi T, Chand T, Strauss L, Poutanen M, Rokka A, Corthals GL, Elo LL. Optimization of statistical methods impact on quantitative proteomics data. J Proteome Res. 2015;14(10):4118–26.
Wu MC, Lee S, Cai T, Li Y, Boehnke M, Lin X. Rare-variant association testing for sequencing data with the sequence kernel association test. Am J Hum Genet. 2011;89(1):82–93.
Ben Mousa A. Sorafenib in the treatment of advanced hepatocellular carcinoma. Saudi J Gastroenterol. 2008;14(1):40–2.
Kim H, Yu SJ, Yeo I, Cho YY, Lee DH, Cho Y, Cho EJ, Lee JH, Kim YJ, Lee S, et al. Prediction of response to Sorafenib in hepatocellular carcinoma: a putative marker panel by multiple reaction monitoring-mass spectrometry (MRM-MS). Mol Cell Proteomics. 2017;16(7):1312–23.
Zhang Y, Yang C, Wang S, Chen T, Li M, Wang X, Li D, Wang K, Ma J, Wu S. LiverAtlas: a unique integrated knowledge database for systems-level research of liver and hepatic disease. Liver Int. 2013;33(8):1239–48.
Camacho JC, Kokabi N, Xing M, Prajapati HJ, El-Rayes B, Kim HS. Modified response evaluation criteria in solid tumors and European Association for the Study of the liver criteria using delayed-phase imaging at an early time point predict survival in patients with unresectable intrahepatic cholangiocarcinoma following yttrium-90 radioembolization. J Vasc Interv Radiol. 2014;25(2):256–65.
Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549.
Akaike H. Information theory and an extension of the maximum likelihood principle. In: Selected papers of Hirotugu Akaike. New York: Springer; 1998. p. 199–213.
Bruix J, Sherman M. Management of hepatocellular carcinoma: an update. Hepatology. 2011;53(3):1020–2.
Ronot M, Bouattour M, Wassermann J, Bruno O, Dreyer C, Larroque B, Castera L, Vilgrain V, Belghiti J, Raymond E. Alternative response criteria (Choi, European association for the study of the liver, and modified response evaluation criteria in solid tumors [RECIST]) versus RECIST 1.1 in patients with advanced hepatocellular carcinoma treated with sorafenib. Oncologist. 2014;19(4):394–402.
Neo SY, Leow CK, Vega VB, Long PM, Islam AF, Lai P, Liu ET, Ren EC. Identification of discriminators of hepatoma by gene expression profiling using a minimal dataset approach. Hepatology. 2004;39(4):944–53.
Qi X, Ng K, Lian QZ, Liu XB, Li CX, Geng W, Ling CC, Ma YY, Yeung WH, Tu WW. Clinical significance and therapeutic value of glutathione peroxidase 3 (GPx3) in hepatocellular carcinoma. Oncotarget. 2014;5(22):11103–20.
Kanda M, Nomoto S, Okamura Y, Hayashi M, Hishida M, Fujii T, Nishikawa Y, Sugimoto H, Takeda S, Nakao A. Promoter hypermethylation of fibulin 1 gene is associated with tumor progression in hepatocellular carcinoma. Mol Carcinog. 2011;50(8):571–9.
Lukanova A, Becker S, Hüsing A, Schock H, Fedirko V, Trepo E, Trichopoulou A, Bamia C, Lagiou P, Benetou V. Prediagnostic plasma testosterone, sex hormone-binding globulin, IGF-I and hepatocellular carcinoma: etiological factors or risk markers? Int J Cancer. 2014;134(1):164–73.
Wu C-C, Hsu C-W, Chen C-D, Yu C-J, Chang K-P, Tai D-I, Liu H-P, Su W-H, Chang Y-S, Yu J-S. Candidate serological biomarkers for cancer identified from the secretomes of 23 cancer cell lines and the human protein atlas. Mol Cell Proteomics. 2010;9(6):1100–17.
Luk JM, Lam BY, Lee NP, Ho DW, Sham PC, Chen L, Peng J, Leng X, Day PJ, Fan S-T. Artificial neural networks and decision tree model analysis of liver cancer proteomes. Biochem Biophys Res Commun. 2007;361(1):68–73.
Ang IL, Poon TC, Lai PB, Chan AT, Ngai S-M, Hui AY, Johnson PJ, Sung JJ. Study of serum haptoglobin and its glycoforms in the diagnosis of hepatocellular carcinoma: a glycoproteomic approach. J Proteome Res. 2006;5(10):2691–700.
Riener MO, Fritzsche FR, Soll C, Pestalozzi BC, Probst-Hensch N, Clavien PA, Jochum W, Soltermann A, Moch H, Kristiansen G. Expression of the extracellular matrix protein periostin in liver tumours and bile duct carcinomas. Histopathology. 2010;56(5):600–6.
Wang X, Chen Y, Qb H, Cy C, Wang H, Liu Z, CHk C, Yew DT, Lin M, He M. Proteomic identification of molecular targets of gambogic acid: role of stathmin in hepatocellular carcinoma. Proteomics. 2009;9(2):242–53.
Gangadharan B, Antrobus R, Dwek RA, Zitzmann N. Novel serum biomarker candidates for liver fibrosis in hepatitis C patients. Clin Chem. 2007;53(10):1792–9.
Graveel CR, Jatkoe T, Madore SJ, Holt AL, Farnham PJ. Expression profiling and identification of novel genes in hepatocellular carcinomas. Oncogene. 2001;20(21):2704–12.
Chung Y, Rabe-Hesketh S, Choi IH. Avoiding zero between-study variance estimates in random-effects meta-analysis. Stat Med. 2013;32(23):4071–89.
This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI16C2037, HI15C2165). Publication of this article was sponsored by HI16C2037 grant.
This article has been published as part of BMC Systems Biology Volume 12 Supplement 9, 2018: Proceedings of the 29th International Conference on Genome Informatics (GIW 2018): systems biology. The full contents of the supplement are available online at https://bmcsystbiol.biomedcentral.com/articles/supplements/volume-12-supplement-9.
Department of Statistics, Seoul National University, Seoul, South Korea
Jongsu Jun
, Yongkang Kim
& Taesung Park
Graduate School of Public Health, Seoul National University, Seoul, South Korea
Jungsoo Gim
Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, South Korea
Hyunsoo Kim
, Injun Yeo
, Jiyoung Park
& Youngsoo Kim
Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University College of Medicine, Seoul, South Korea
Department of Internal Medicine and Liver Research Institute, Seoul National University, Seoul, South Korea
Su Jong Yu
, Jeong-Ju Yoo
, Young Youn Cho
, Dong Hyeon Lee
, Eun Ju Cho
, Jeong-Hoon Lee
, Yoon Jun Kim
& Jung-Hwan Yoon
Department of Mathematics and Statistics, Sejong University, Seoul, South Korea
Seungyeoun Lee
Interdisciplinary program in Bioinformatics, Seoul National University, Seoul, South Korea
Taesung Park
Search for Jongsu Jun in:
Search for Jungsoo Gim in:
Search for Yongkang Kim in:
Search for Hyunsoo Kim in:
Search for Su Jong Yu in:
Search for Injun Yeo in:
Search for Jiyoung Park in:
Search for Jeong-Ju Yoo in:
Search for Young Youn Cho in:
Search for Dong Hyeon Lee in:
Search for Eun Ju Cho in:
Search for Jeong-Hoon Lee in:
Search for Yoon Jun Kim in:
Search for Seungyeoun Lee in:
Search for Jung-Hwan Yoon in:
Search for Youngsoo Kim in:
Search for Taesung Park in:
JJ and TP developed statistical method. JJ performed statistical analysis. HK, SJY, EJC, J-H Y, YSK and TP conceived and planned the experiments. HK, JP, and J-JY carried out the experiments. JJ, JG. YK and TP planned and carried out the simulations. YYC, DHL, YC, EJC, J-HL, and YJK contributed to sample preparation. JJ, JG, YK, YSK, TP contributed to the interpretation of the results. JJ and TP took the lead in writing the manuscript. All authors have read and approved the final manuscript.
Correspondence to Taesung Park.
To identify candidate serum hepatocellular carcinoma (HCC) biomarkers for prognosis and response to the tyrosine kinase inhibitor sorafenib, data from 115 HCC patients who had undergone continuous administration of sorafenib for more than 6 weeks, were collected from Seoul National University Hospital, as part of an ongoing study between May 2013 and August 2014. HCC was diagnosed by histological or radiological evaluation, with reference to the American Association for the Study of Liver Diseases (AASLD) [31] or the European Association for the Study of the Liver (EASL) [32] guidelines. All procedures/analyses were approved by the Seoul National University Hospital Institutional Review Board (IRB protocol No. 0506–150-005).
Jun, J., Gim, J., Kim, Y. et al. Analysis of significant protein abundance from multiple reaction-monitoring data. BMC Syst Biol 12, 123 (2018) doi:10.1186/s12918-018-0656-9
Multiple reaction-monitoring (MRM)
Logistic regression-based method for significance analysis of multiple reaction monitoring (LR-SAM)
Sorafenib response


text:
Q: Java efficiency I'm playing with some piece of code calculating the time needed to compute some Java code to get a feeling of the efficiency or inefficiency of some of Java's functionality. Doing so I'm stuck now with some really strange effect I just can't explain myself. Maybe someone of you can help me understand it.
public class PerformanceCheck {

 public static void main(String[] args) {
    List<PerformanceCheck> removeList = new LinkedList<PerformanceCheck>();

    int maxTimes = 1000000000;

    for (int i=0;i<10;i++) {
        long time = System.currentTimeMillis();

        for (int times=0;times<maxTimes;times++) {
            // PERFORMANCE CHECK BLOCK START

            if (removeList.size() > 0) {
                testFunc(3);
            }

            // PERFORMANCE CHECK BLOCK END
        }

        long timeNow = System.currentTimeMillis();
        System.out.println("time: " + (timeNow - time));
    }
 }

 private static boolean testFunc(int test) {
    return 5 > test;
 }

}

Starting this results in a relatively long computation time (remember removeList is empty, so testFunc is not even called):
time: 2328
time: 2223
...

While replacing anything of the combination of removeList.size() > 0 and testFunc(3) with anything else has better results. For example:
...
if (removeList.size() == 0) {
    testFunc(3);
}
...

Results in (testFunc is called every single time):
time: 8
time: 7
time: 0
time: 0

Even calling both functions independent from each other results in the lower computation time:
...
if (removeList.size() == 0);
    testFunc(3);
...

Result:
time: 6
time: 5
time: 0
time: 0
...

Only this particular combination in my initial example takes so long. This is irritating me and I'd really like to understand it. What's so special about it?
Thanks.
Addition:
Changing testFunc() in the first example
if (removeList.size() > 0) {
                testFunc(times);
}

to something else, like
private static int testFunc2(int test) {
    return 5*test;
}

Will result in being fast again.

A: That is really surprising. The generated bytecode is identical except for the conditional, which is ifle vs ifne.
The results are much more sensible if you turn off the JIT with -Xint. The second version is 2x slower. So it's to do with what the JIT optimization.
I assume that it can optimize out the check in the second case but not the first (for whatever reason). Even though it means it does the work of the function, missing that conditional makes things much faster. It avoids pipeline stalls and all that.

A: While not directly related to this question, this is how you would correctly micro benchmark the code using Caliper.  Below is a modified version of your code so that it will run with Caliper.  The inner loops had to be modified some so that the VM will not optimize them out.  It is surprisingly smart at realizing nothing was happening.
There are also a lot of nuances when benchmarking Java code.  I wrote about some of the issues I ran into at Java Matrix Benchmark, such as how past history can effect current results.  You will avoid many of those issues by using Caliper.

*

*http://code.google.com/p/caliper/

*Benchmarking issues with Java Matrix Benchmark
public class PerformanceCheck extends SimpleBenchmark {

public int timeFirstCase(int reps) {
    List<PerformanceCheck> removeList = new LinkedList<PerformanceCheck>();
    removeList.add( new PerformanceCheck());
    int ret = 0;

    for( int i = 0; i < reps; i++ )  {
        if (removeList.size() > 0) {
            if( testFunc(i) )
                ret++;
        }
    }

    return ret;
}

public int timeSecondCase(int reps) {
    List<PerformanceCheck> removeList = new LinkedList<PerformanceCheck>();
    removeList.add( new PerformanceCheck());
    int ret = 0;

    for( int i = 0; i < reps; i++ )  {
        if (removeList.size() == 0) {
            if( testFunc(i) )
                ret++;
        }
    }

    return ret;
}

private static boolean testFunc(int test) {
    return 5 > test;
}

public static void main(String[] args) {
    Runner.main(PerformanceCheck.class, args);
}
}

OUTPUT:
 0% Scenario{vm=java, trial=0, benchmark=FirstCase} 0.60 ns; σ=0.00 ns @ 3 trials
50% Scenario{vm=java, trial=0, benchmark=SecondCase} 1.92 ns; σ=0.22 ns @ 10 trials

 benchmark    ns linear runtime
 FirstCase 0.598 =========
SecondCase 1.925 ==============================

vm: java
trial: 0

A: Well, I am glad not having to deal with Java performance optimizations. I tried it myself with Java JDK 7 64-Bit. The results are arbitrary ;). It makes no difference which lists I am using or if I cache the result of size() before entering the loop. Also entirely wiping out the test function makes almost no difference (so it can't be a branch prediction hit either).
Optimization flags improve performance but are as arbitrary.
The only logical consequence here is that the JIT compiler sometimes is able to optimize away the statement (which is not that hard to be true), but it seems rather arbitrary. One of the many reasons why I prefer languages like C++, where the behaviour is at least deterministic, even if it is sometimes arbitrary.
BTW in the latest Eclipse, like it always was on Windows, running this code via IDE "Run" (no debug) is 10 times slower than running it from console, so much about that...

A: When the runtime compiler can figure out testFunc evaluates to a constant, I believe it does not evaluate the loop, which explains the speedup.
When the condition is removeList.size() == 0 the function testFunc(3) gets evaluated to a constant.  When the condition is removeList.size() != 0 the inner code never gets evaluated so it can't be sped-up. You can modify your code as follows:
for (int times = 0; times < maxTimes; times++) {
            testFunc();  // Removing this call makes the code slow again!
            if (removeList.size() != 0) {
                testFunc();
            }
        }

private static boolean testFunc() {
    return testFunc(3);
}

When testFunc() is not initially called, the runtime compiler does not realize that testFunc() evaluates to a constant, so it cannot optimize the loop.
Certain functions like
private static int testFunc2(int test) {
    return 5*test;
}

the compiler likely tries to pre-optimize (before execution), but apparently not for the case of an parameter is passed in as an integer and evaluated in a conditional.
Your benchmark returns times like
time: 107
time: 106
time: 0
time: 0
...

suggesting that it takes 2 iterations of the outer-loop for the runtime compiler to finish optimizing.  Compiling with the -server flag would probably return all 0's in the benchmark.

A: The times are unrealistically fast per iteration.  This means the JIT has detected that your code doesn't do anything and has eliminated it.  Subtle changes can confuse the JIT and it can't determine the code doesn't do anything and it takes some time.
If you change the test to do something marginally useful, the difference will disappear.

A: These benchmarks are tough since compilers are so darned smart.  One guess:  Since the result of testFunc() is ignored, the compiler might be completely optimizing it out.  Add a counter, something like
   if (testFunc(3))
     counter++;

And, just for thoroughness, do a System.out.println(counter) at the end.

redpajama_set_name: RedPajamaStackExchange


Test all Links before you commence sending the email, and |} At case the new member is downloading something, ensure the download procedure is frictionless and hassle-free. Casual emails have a tendency to have a better reply rate, and it is always suggested that you write like you're addressing a recipient personally. Nonetheless their emails are quite crucial for all of us. However great your email is, the most important purpose of the customer is to quit speaking to you. Response to your emailseven in case the email was not meant for you It's tricky to answer to every email message ever sent to you, but you need to try to. Writing successful advertising emails is a tiny Goldilocks-esque.
You have to constantly be innovating and iterating with every single Every email. An email is an instance of an interactive composing, meaning that we are writing to someone instead of only for someone to read. Put the cancellation period (the amount of time you need to determine if you'd like to unsend an email). It's simple to misunderstand the significance of an email and possibly get insulted. You understand pretty well that email was ignored. When it's a really first email, it needs to be an extremely low-cost ask, too.
As you Probably have several folks sending emails on behalf Of your company, it is critical to make their email signatures seem the specific same. Your email is most probably neither for them, so you must make the price of responding essentially zero. Apparently, you should send an eloquent email speaking about how you are able to help your potential and nail your value proposition. Just take the chance to read the entire email before you respond.
If you buy a lot of mails and you fight to remain organized, Put an auto responder in your email that lets folks know that it may be two or three times ahead of your respond. Unlike what you might have heard, email is not always a sort of informal correspondence. Keep in mind, however, that some folks are ticked away by emails that are overly direct. It's difficult to achieve, that's why I will demonstrate how to write a powerful sales introduction email. Many don't want their email address to be distributed to individuals they do not know.
Creating a Website for your engraving business is a great means to Receive clients. In an official letter you're typically writing to obtain or provide details about company, school, employment, etc.. All info is left justified for any letterhead style. Obviously, you understand that you have got to provide individual contact information, for example email, mobile phone number, and probably your LinkedIn profile.
You want to do a lot of promotion for your engraving business. If you're so imaginative, laser engraving business is appropriate for you. A laser engraving firm is an extremely profitable business you can start from your residence. Starting a laser engraving company is very rewarding. Do not make the customer feel like you didn't read their whole email. You will find quite a great deal of products it is possible to make and place on the business.
While the remainder of your email is centered on what you could Do to help your potential and the way that your merchandise helps them. All these are capable of accomplishing what it is that you're doing, but they are not required to achieve that. There is no need to incorporate the recipient's first name unless you aren't sure of the individual's gender. Finding out how to craft specialist emails is an ability you may take with you in the so-called actual world. There are additional opportunities. Success in any company demands good preparation.
Make certain it matches the character of your job or it is even Pertinent to the typical recipients and content of your email. There's maybe a way in which suffering can be considered a mystery when it is not likely to be directly brought on by the devil, man and God. Perhaps you'll receive a notion of manners which you can better market yourself, you can meet somebody you can begin a new business with, and you're going to get notifications on industry-related workshops and socials.
If you're sending your cover letter' within an email, then it's Recommended to produce the cover letter the significant body of the email and just attach your CV. Your company letter has to be suitably typed and well spaced A work program is an official document. A nice small business letter shouldn't be more than 1 page. Do not use fancy fonts and colours Always remember a letter is a formal company letter.
If you do not understand what your email signature needs to look Enjoy, begin with the main information only. An extremely easy email signature is only one of the most frequent elements of the online etiquette. The crucial point to keep in mind is that an email signature is not the purpose of a conversation.
A memorandum might be utilised to earn a request, declare an Approaching meeting, or communicate a few vital facts. A memorandum of agreement clarifies the obligations and responsibilities of person parties with respect to one another and also with regard to the undertaking or assignment. Internal Memorandum Is just like the company memorandum except with the benefit of being just read in a particular workplace only, not the entire enterprise. In investment fund, an offering memorandum is a sort of an exhaustive company application that highlights data required by means of an investor to know the company. A memorandum is also essential in the area of business as it assists build excellent relationships. As an example, a Memorandum of Understanding will pay for the kinds of insurance that the parties have, such as liability insurance, and the promises everyone is ready to make and everybody's degree of commitment to the project accessible. You don't have to sign the memorandum facing witnesses because you would a will.
An MOU is often the first step in creating a legally Binding contract. An MOU is an easy understanding between two parties without a transfer of products, services, or funds. However iron-clad an MOU could seem, however, the parties involved must still check with an attorney before proceeding with the conditions of this arrangement. An MOU should not be tried if it is going to disrupt a great relationship.
Compile all the information that you want to include in the Contract in 1 document for the reference. An executory agreement is a contract that hasn't yet been fulfilled and that's known to be one that will be fulfilled at a subsequent date. An executed agreement is an agreement that has been signed by all the parties involved.
If your goal is to chat about a new company policy, utilize a strict tone. If it's to inform your co-workers of the approaching company party, go gently. Once the objective of the memo has been established the remainder of the memo ought to be a succinct summary of the facts applicable to the function of the memo. The intent of a memo is to share information in an office.
A memo letter might still be sent by email. Effective letters will Yield only excellent outcomes. Letters of intent exist to prevent conflicts that may appear from disagreements afterwards.
Memos want to convey data in a concise, purposeful way. They Are generally considered a formal method of communicating. Furthermore, a memo is not normally a place for editorializing. The memo, as among the most frequent types of internal small business communication in virtually any sort of organization, is surely a worthwhile format to comprehend.
Memos are still an essential to each company or organization. An Internal memo could possibly be a better option, especially if everything you have to say is vital. Memos also function as evidence in case of disputes between employees and supervisors. They save time as they can be read at an employees own time. You do not have to record memos in the reference list based on the American Psychological Association.
Presenting a memo will then show seriousness from the Professionalism and company in the business enterprise. It is primarily a communication tool used in business in disseminating information throughout the organization. Even though it may be composed to some individuals or with numerous audiences in mind, typically it is a highly goal-oriented communication between two individuals who need to share info. By way of instance, it was given to several sections of a certain university, the memorandum depicting the exact same information with the goal of notifying the various departments of the memo. As it's effectively an internal news thing informing others of specific developments, it's important to get a clear idea of the information that needs to be communicated. When it's an official memo that is issued, the very first area of the paragraph must state the function of the memorandum and after that proceed to enlist extra info. Some folks seem to think that memos, since they're public, are powerful management tools.
The memos are often sent out to lender customers together with Their monthly bank statements. As with other small business communications, a memo asking advice gives a compact overview of the issue you're addressing. Memos are usually sent to numerous people at one time and are not meant to be personalized just like a business letter is. Company memos these days are written to be short and concise in order for it to be considered a successful memo.
After you inform us about All the paper information, we will Start searching for a suitable writer for your paper. You can write about a few personal details that you have never told anybody. You will only have to give us all the details and information you want to add in the newspaper, in addition to the instructions supplied by your professor.
Nearly all of examples are posted as a member of composing guides Published online by educational institutions. Our high quality autobiography they can offer you with a fantastic outline, as well as, ideas to compose a superb autobiography. You're welcome to use this example to make your own A-Level pupil autobiography! Just keep reading and you'll understand what we mean after checking out our entirely free example. You can look for short autobiography examples if you are still confused about writing it. So, this really is our simple example of a pupil autobiography which you are able to use while creating your personal memoir writing.
The specific same is true when it comes life stories. Conclusion Here you have to highlight the principal facts that you want your reader to listen to, to underline the main things about you. Reading autobiographies might be more intriguing than biographies because you are studying the thoughts of the individual rather than somebody else's interpretation. You are not predicted to be somewhat creative.
Spiritual and spiritual experiences are extremely private. If You believe nothing is intriguing about your life, then you're wrong. Unfortunately, a great deal of people don't have happy, shining lives. Somebody works part-time and doesn't have enough time to perform all of the assignments. You can attain this in a few ways. My mom claims that my first word was cash.
If you've already graduated from university or college and are Searching for an excellent job, you want to acquire a persuasive resume to impress your future employer. Every student needs help with homework from time to time. Fully being a happy high school student would not be possible with no loved ones. There are instances some educators are not professional or do not explain quite a few of the lessons well. Apart from that, the instructor also told my parents I was too small scared I was not able to catch up.
Some folks could have their own life story written by another person Since they don't think they could write well, but they're still considered an author as they're providing the advice. Added it is the simplest way to begin any story about any person. If you discover that the author didn't provide precisely what you anticipated, request a revision, and we'll make the corrections. You might also get in touch with your author to provide several additional recommendations or ask information regarding the order's progress. The authors have to learn what their intentions must have the ability to write in their lifetime, and they then can choose the best form to educate their own tale. The exceptionally gifted writers have all the vital skills to create superior-quality content.
Utilize the Help of true academic specialists and receive the Service you have earned! Tell our specialists what kind of homework help on the internet you wish to get. You might discover more details. Below are a couple of Autobiography OutlineExamples to your use. Consequently, if you want to locate this amazing picture approximately ten tons Of Autobiography for Pupils, just click on save button to conserve this images to your PC. A biography can be a means to divide time. It's different in the biography, that is the life story of someone written by someone else.
There are many kinds of autobiographies. They are also significant Because they allow other individuals in similar circumstances understand they are not alone. Childhood If you don't know what things to start your autobiography with, it is quite logical to begin to your early years. Autobiographies are an essential part of history. The autobiography is a significant area of the film. Autobiographies are likewise a significant part of history. It has to be intriguing it is very clear that you need to make the autobiography intriguing to see.
It's possible to observe different autography outline template To get a very clear image of the way to create it. Possessing a principal Theme can allow you to bring everything together and maintain the attention of the Readers from the start to the finish of the autobiography. Everything is Dependent upon the means of presentation. Academic papers cannot contain any Signs of plagiarism. The Most Often encountered newspaper writing service The bulk of our clients require is informative writing. Nowadays it's quite hard To locate a trusted essay writing service. If it's the very first time You're very likely to use our article writing service, you most likely have lots of questions.


Cattleya and Laelia photos | Southern Suburbs Orchid Society Inc.
This photo gallery has some examples of the many varieties of cattleyas and laelias our members grow and show at either our monthly meetings; inter-club contests; State Shows or at our Annual Spring Show.
If you are interested in handy tips on growing Cattleyas and Laelias, visit our Orchid Care -Cattleya and Orchid Care-Laelia pages.


your code and check whether your discount was reflected and continue your checkout. Updated on August 2,2018. Your cooperation and understanding will be appreciated.
By creating an account with our store, you will.
Try, progenex discount codes if one is not.
Progenex was created to help you reach your fitness goals faster.
To provide you a better service and protect the security of our website, we will check all the access requests. Due to the abnormal access request from the IP address you used, we need your help to verify that you are human. Progenex Promo Code Deal last updated on July 31, 2018.
Z Categories, about US, contact US 2018 Dealscove coupons get coupon code for 62,981 stores. Copy 50 say worked. Step 3: Scan your shopping cart to confirm that the coupon code was properly entered and is reflected in your total price. If you have any questions about using your coupons online, please contact m Customer Service. People can find numerous options online to consider and shop at progenex, using online coupon codes and discounts.


text:
by MacHelpAdmin 05 April 2018
According to recent rumors, the upcoming iOS 12 will rather focus on performance and stability improvements than on adding new features. Among the novelties, there is the possibility to activate animoji in FaceTime, deeper integration of Siri with the iPhone, and some new possibilities for the Do Not Disturb mode. The fact that a new iOS lacks smart new features may be explained by two reasons. First, such novelties are likely to come with iPhone update. And the second reason consists in the disturbingly big number of bugs in the current iOS, which makes Apple concentrate their efforts on quality and reliability rather than novelties. The second reason looks especially convincing, as the company has been criticized for both security and quality issues in the previous iOS updates.
Anyway, Apple's intention to delay the release of features that are not sufficiently polished means that we should not expect from the new update too many new goodies. At least the novelties in the iOS 12 will be considerably fewer than the iOS 11 changes. There is even a list of features that Apple has postponed until the next iOS. The list of features that will NOT appear before 2019 includes the following items:
The long-expected change of the icon-based home screen.
Redesign of the CarPlay interface.
Changes for Mail, updates for picture taking and photo editing.
Updates to Camera app and Photos.
The old home screen is likely to stay in the iOS 12
What's New Anyway?
But in any case a new iOS simply cannot come without nice novelties; there must be at least a couple of them. That's right, and even iOS 12 does introduce certain new features, though fewer than was initially planned (and expected).
Reports say the improvements will focus on "augmented reality, digital health, and parental control". As regards digital health, a few new important features (like increased availability of patients' medical information) were added to the Health application already with the iOS 11.3. A capability to map vertical surfaces was introduced in the ARKit, together with some minor boons. The iOS 12 is likely to further develop the two above guidelines. As to parental control, recently there has been frequent criticism from important figures and organizations, including some major Apple shareholders, with insufficient elements of such control at iOS level. Public urges the company to make it easier for parents to restrict children's use of smartphones. As a response, Apple announces its further steps to advance in this direction as well. The company's intention is to enhance the iOS abilities that enable parents to control their children's use of smartphones and help parents restrict what their kids consume from their digital devices.
Among other expected changes in the iOS 12 is the possibility for the users to place Animoji faces over themselves while making video calls (really amusing!). Another novelty: Siri will be deeper integrated with the iPhone in the search view, which means more help from the smart assistant during the use of the system-wide search capacity. As regards the Do Not Disturb mode, additional options are expected for automatic rejection of phone calls and silence notifications.
ALSO iOS 11 The Most Dramatic and Fantastic Upgrade in Years
Improved Stability
Some of the stability issues the iOS 12 is expected to address are just annoyances, like the auto-correct error on the keyboard. Others bring more serious consequences like shutdown or dangerous vulnerabilities as the December 2 bug or the Spectre and Meltdown phenomena do. An unpleasant surprise for the users was undoubtedly the iPhone crash on receiving a particular iMessage. Overall, Apple seems to undertake a serious strategic shift towards greater reliability and security of its products, which is reflected even in the fact that the upcoming iOS 12 is code-named as Peace. Reportedly, the company's new guideline is to focus on fewer and smaller new features, while ensuring a stricter safety and stability of its operating systems.
Fewer Smarties, Tougher Sells
It is clear, however, that users are much stronger attracted by bright novelties that can be enjoyed just upon purchase, than by reassurances of improved stability. Therefore, the current shift in Apple's policy may have mixed consequences in terms of sales volume and probably the company's profits. But, on the other hand, users also care about security and quality of their purchases and are not solely driven by the desire to try a new goody. Future will show how wise the current policy is.
The release date is still unknown, but there are no reasons for this year's timing to be different from usual. So, unless some major stray from the common schedule occurs, the overall sequence of events related to the release of the next iOS will be as follows:
Release announcement and presentation of major features of the iOS 12 at WWDC (most likely in June).
Seeding the beta versions for testing (first to developers and then to the public) starts around WWDC date.
Final release, likely to take place in September, shortly before the release of a new bunch of iPhone models.
How to Get iOS 12?
You can't buy it before the official release, but in principle, you can try the new iOS earlier. Developers and participants of public beta testing will have access to it shortly after WWDC this summer (first the developers). Developers are welcome to Apple Developer site and those taking part in public testing – to beta.apple.com, as soon as the corresponding testing stages are announced. While it is not as simple to become a developer, a regular user may easily become a regular tester. For this to be done, you download a special profile, which will enable you to join the process.
General Instruction on Cache Management in Your Mac

redpajama_set_name: RedPajamaCommonCrawl


Visit us and have a great experience when you start looking for a car at Carls Auto Sales of Walker. When you drive off our lot, your vehicle and payment will have exceeded your expectations. Our experienced sales associates are here to get you the ideal vehicle. With lots of vehicles to choose from, we will find the right car or truck to match your budget. All of your used vehicle needs can be handled here at Carls Auto Sales. Anything you may need in a used car, Carls Auto Sales is here to help. So come check out Carls Auto Sales of Walker at 13635 Burgess Ave, Walker, Louisiana 70785 and let us help you get the car you've always wanted.


text:
Posts Tagged 'Congressman Glenn "GT" Thompson'
Congressional CTE Caucus Holds Briefing on Increasing Access and Equity
On Wednesday, November 20, the Congressional Career and Technical Education (CTE) Caucus held a briefing on Increasing Access and Equity in Career & Technical Education. During the briefing, panelists discussed challenges to increasing access and equity in CTE, ongoing efforts to address gaps and how the Strengthening Career and Technical Education for the 21st Century Act (Perkins V) can help to close equity gaps.
Co-Chair of the Congressional CTE Caucus, Congressman Glenn "G.T." Thompson (R-PA), joined the session and gave remarks to the group. Congressman Thompson voiced his support for high quality CTE programs, and expressed the urgent need to improve access and equity to and within these programs. He shared that "the power of work must be accessible to everyone, and CTE offers significant opportunities."
The full panel represented the state, local and employer perspectives, and included:
David Abromowitz, Chief Public Policy Officer, YouthBuild USA
John LeTourneau, Director of Manufacturing, Anoka area chamber of Commerce
Brianna McCain, Policy Associate, Advance CTE (Moderator)
Nina Salomon, Deputy Program Director, Corrections & Reentry, Council of State Governments
Spencer Sherman, Director, Office of College and Career Readiness, Rhode Island Department of Education
Advance CTE's Brianna McCain started off the briefing by discussing the history and current trends of access and equity in CTE, as well as how states can leverage Perkins V to support high quality CTE for each and every learner. Brianna pointed out that as the quality of CTE programs has significantly improved, the nature of the equity challenge in CTE has changed. Today, a renewed focus must be placed on ensuring equitable access to these high-quality programs.
All panelists agreed that it is essential to collaborate across agencies and with employer and community representatives. This is the way to ensure that all populations are being reached, and that programs are developed to set learners up for success. When speaking about the juvenile justice population, Nina Salomon shared how she found that most juvenile justice agencies are not partnering with workforce agencies, education agencies or employers. This led to a disconnect in all groups understanding what resources are available, including federal funding opportunities through Perkins and WIOA. Therefore, administrators have been missing out on opportunities to provide education and training in juvenile justice systems.
Check out Advance CTE's Making Good on the Promise series to learn more about how states can leverage data to identify and address equity gaps, rebuild trust with historically underserved communities, expand access to high-quality CTE for each and every learner and build systems to ensure learner success.
Meredith Hills, Policy Associate
By Meredith Hills in Uncategorized
Tags: Access and Equity, Congressional CTE Caucus, Congressman Glenn "GT" Thompson, juvenile justice, Perkins V
Tackling Rural CTE Challenges on Capitol Hill
What can state leaders do to expand access to high-quality career technical education (CTE) in rural communities? That was the focus of an event held last Thursday by the Congressional CTE Caucus, in coordination with Advance CTE.
The event featured state and local leaders from diverse geographies, who shared their experiences with delivering CTE in rural communities, highlighting both barriers and best practices. The co-chairs of the Congressional CTE Caucus, Representatives Glenn "GT" Thompson (R-PA) and Jim Langevin (D-RI), also stopped by the event to provide remarks about the value of CTE and weigh in on the state of rural CTE.
As Representative Thompson (R-PA), who comes from a largely rural area in Pennsylvania, noted at the beginning of the event, "CTE restores rungs on the ladder of opportunity. We need to make sure that we are dealing with the barriers," so that CTE can help close opportunity gaps and extend a bridge to lifelong career success. However, rural communities often face obstacles like scarce resources, critical teacher shortages and a limited employer base that make it difficult to deliver high-quality CTE at scale. As Advance CTE found through their interviews with state CTE leaders, these challenges are common across geographies, yet they are often exacerbated in rural communities (find all of the briefs in the CTE on the Frontier series here).
How can state and local leaders tackle these challenges? During the event, Dr. Marcie Mack — state CTE director for Oklahoma, spoke about Oklahoma's career-tech system and the value of employer partnerships in rural CTE, particularly with Oklahoma's technology center districts, which deliver CTE programs to high school, postsecondary, adult and justice-involved students.
We were pleased to also feature on the panel a 2018 Excellence in Action award winner, Traverse Bay Area Intermediate School District Career-Tech Center. Stephanie Long, curriculum supervisor for TBAISD, shared about the difficulty of connecting learners from rural districts with the high-quality CTE programming her career-tech center provides. The center serves a region as large as the state of Delaware, with many students traveling hours a day to get to and from classes. The center provides buses during the day and bus passes after school hours to help these learners access high-quality CTE.
Jan Hanlon, executive director for the Mountain State Education Service Cooperative, discussed how West Virginia is tackling access challenges through Simulated Workplace programs. Through Simulated Workplace, more than 24,000 students annually develop real-world skills by participating in a realistic classroom-based company where they have to meet expectations for attendance, safety, sobriety and professionalism, just as they would need if employed by a local business. Representative Langevin (D-RI) of Rhode Island wrapped up the discussion with an important closing thought: "We can't expect businesses to grow if we don't have the workforce available."
As the event's panelists underscored, CTE is a critical strategy to help rural America adapt to the 21st century economy. The aftershocks of the Great Recession are still being felt in rural America today, where many learners are disconnected from opportunities to reskill and prepare for the jobs of tomorrow. CTE can help these learners build the skills they need for lifelong success and equip them with the knowledge and abilities to adapt to an ever-changing economy.
By Austin Estes in Meetings and Events
Tags: Congressman Glenn "GT" Thompson, Congressman Jim Langevin, Michigan, Oklahoma, rural, Simulated Workplace, Traverse Bay, West Virginia


text:
Q: On iOS, can two Timers be set at the same time, or have a timer fire off an event 0.0083 second later to achieve 120 fps? It seems like the highest timer rate on iOS is 60 times per second using CADisplayLink, which is 0.01666 second each time.  Can two timers be set up at the same time, so that an image can be updated more frequently for its location?
Or, can one timer event handler fire off another timer event 0.00833 second later, so that there is another update in between the 1/60 intervals, to achieve 120 fps?

A: Why would you want to achieve 120 fps on a screen that updates 60 times a second?
That would just waste CPU and power. Don't do it.
However, having a timer that fires more often is definitely possible.
Drawing in each of these callbacks is, however, is not possible. (at least using -drawRect:, don't know about OpenGL).

A: You can't do this for a number of reasons. First, there's a hard cap at 60fps due to hardware limitations. Second, NSTimer is not nearly accurate enough for this sort of thing. From the Timer Programming Topics - Timer documentation (bold added for emphasis):

Timing Accuracy
A timer is not a real-time mechanism; it fires only when one of the
  run loop modes to which the timer has been added is running and able
  to check if the timer's firing time has passed. Because of the various
  input sources a typical run loop manages, the effective resolution of
  the time interval for a timer is limited to on the order of 50-100
  milliseconds. If a timer's firing time occurs while the run loop is in
  a mode that is not monitoring the timer or during a long callout, the
  timer does not fire until the next time the run loop checks the timer.
  Therefore, the actual time at which the timer fires potentially can be
  a significant period of time after the scheduled firing time.


text:
EndNote reverse-engineering case looks headed to courtroom
Thomson Reuters, publisher of EndNote, is suing George Mason University over …
Jonathan M. Gitlin - Nov 4, 2008 1:35 pm UTC
As anyone who works in academia knows, writing and publishing papers involves frequently citing the existing literature. When you're working on a paper with 30 or more references, keeping track of them all can be a downright pain, which is where reference-managing software like Thomson Reuters' EndNote comes in. EndNote is the market leader in this field, but recently it has been facing competition from the open source Zotero, which is a Firefox plugin that lets you manage your bibliographic library and insert references into papers. Right now though, EndNote and Zotero are locked in a legal battle over claims by Thomson Reuters that the developers of Zotero have illegally reverse-engineered aspects of EndNote.
Although EndNote has its position as the 800lb gorilla in the reference manager field, it wouldn't be an exaggeration to say that many users view the application with a measure of antipathy. It can sometimes be frustrating to use, and the company's practice of rolling out updates that can appear to be bug fixes with a $99 price tag has created an environment where other developers have begun to enter the market with competing solutions.
As I noted in my Office 2008 review, Microsoft has inserted a reference manager into the new versions of Word, but where EndNote triumphs over others is the vast breadth of output styles available to users. These are necessary, as individual journals often have subtly different ways that they want citations to be presented; references shown in the text as numbered articles that are then listed in that order in the bibliography, or listed in the text as (author name, year) and then listed alphabetically, for example.
Given that the process of publication can sometimes involve sending the same manuscript to more than one journal, being able to reformat the way the references are presented is an invaluable tool. By virtue of being a market leader with a long history in the field, EndNote is able to offer the widest selection of these output styles as downloads, and by virtue of their large selection of output styles, they remain the market leader.
Zotero is an open source project led by a pair of academics, Dan Cohen and Sean Takats, at George Mason University's Center for History and New Media. Zotero is a plugin for the Firefox browser, and therefore cross-platform, and also has the advantage of being free. It also includes functionality similar to the Mac OS X application Papers, in that it manages PDF libraries, as well as offering users a way to insert references into a document.
The lawsuit, brought by Thomson Reuters against George Mason University and the Comptroller of Virginia, alleges that GMU is in contravention of their EndNote license with their newest version of Zotero, thanks to Zotero having allegedly reverse-engineered the file format that EndNote uses for citation styles in order to offer a similar functionality in Zotero. Thomson Reuters claims that GMU is causing "irreparable harm" to its brand, and is seeking to prevent GMU from distributing the offending application, as well as significant financial damages.
GMU denies this claim, insisting that, although Zotero can read EndNote's .ens files, the application does not convert that data to Zotero's .csl format. GMU has decided not to renew its site license for EndNote, and has re-released the controversial Zotero 1.5 Sync Preview.
For its part, Thomson Reuters remain resolute in its claim that Zotero infringes on their intellectual property and that GMU violated the terms of their site license. According to Thomson Reuters VP Dave Kochalko, "These format files only exist as software code; there is no content or information independent of lines of code and these files can only be interpreted by the computer. A key value of EndNote is its ability to format a bibliography within a manuscript and the format files are integral to that capability."
Kochalko also maintains that "We have worked diligently over the past several months to resolve this matter amicably. Since it has become clear that a resolution is not possible at this time, we have no choice but to pursue litigation in order to protect our intellectual property, as well as protect our bibliographic formatting capability, an important publishing resource our EndNote user-community has relied upon for many years."
As things currently stand, it seems that this case will be heard in front of a court, which will have to decide if GMU really did reverse-engineer EndNote's style files.
Jonathan M. Gitlin Jonathan is the automotive editor at Ars Technica, covering all things car-related. Jonathan lives and works in Washington, D.C.
Email jonathan.gitlin@arstechnica.com // Twitter @drgitlin


Race City Roofing also offers professional & reliable commercial roofing services in Charlotte, NC!
Commercial Roofing Services – Buying a new roof for any building, commercial or residential, is a huge investment, and one you do not want to scrimp on. If you shop for a new roof based solely on price, you will likely take a hit in quality and end up spending more on roofing repairs, maintenance, or even complete roof replacement in the long run. That's why it's so important to choose a commercial roofing contractor who understands your needs and has many years of professional experience.
At Race City Roofing in Mooresville, North Carolina, we have an A+ rating with Angie's List and are a member of the Better Business Bureau and the local chamber of commerce and specialize in commercial roofing services in Charlotte, Mooresville, Cornelius, Huntersville, Denver, Troutman, Statesville and Lake Norman NC.
In addition to our expert commercial roofing repair and installation services, we also offer a lifetime warranty on labor for all of our commercial roofing services we do. We can add a high-quality, weather-proof roof to your church, office building, shopping center, or any other commercial building.
Our commercial roofing contractors use high-quality products, including PremiumCoat roof coatings from Hydro-Stop. With PremiumCoat, your commercial roof will be able to stand strong against almost any surprise, including the fiercest rain, wind, and hail. In fact, your roof will likely last just as long as the rest of the building! That's quality you can't put a price on.
So if you'd like to save thousands of dollars on your commercial roofing project by installing a roof that can survive just about anything, contact us at Race City Roofing. We promise to do our very best to make sure you are satisfied with our work.
Please contact us today to talk about our reliable & affordable commercial roofing services – we'd be delighted to bid on your commercial roofing project!


The basic functionality of your accounting or financial software is at the core of its appeal to users, and part of that functionality --along with a significant share of its visual appeal-- is rooted in your choice of icons. Selecting banking icons that give your product the right appearance while also allowing for maximum efficiency and ease of use isn't always straightforward, but when you choose a banking icons collection as complete and consistently designed as ours, you'll find that nailing this "finishing touch" can actually have a powerful impact on the core of your financial product.
Beautiful, richly-colored accounting and banking graphics from an abacus and scales to piggy banks and credit cards are ready to become a part of your software product's visual identity. Whether you're looking for banking symbols based on people and user groups, money and currencies, security, common functions and practices, or a complex combination therein, you'll find just the right icons for your needs with our extensive set. High-quality matched with modern perspective and smooth rendering makes our bank icons ideal for giving your software the sleek yet instantly recognizable feel it needs to make a great impression on users.
Each icon included in this premium collection features five resolutions, four file types (including ICO, BMP, PNG, and GIF), and three visual states to provide maximum utility and easy integration into any financial application. When you invest in expertly-designed and reliable bank icons, you'll do your project a favor it can use to turn your work into enhanced user experiences and increased profits. Give your application the functionality and visual fluency it needs to become a memorable and meaningful tool with bank icons that know how to produce great results.


Walt Disney: One Man's Dream opened in 2001 at the Studios as a museum and theater honoring Walt's life and work that would lead to what we know as Disney. For the better part of almost two decades, guests could see historical Disney memorabilia like Walt's desk, his original movie cameras, and photographs. The theater at the end of the attraction had, historically, shown a film about Walt Disney.
The last few years, however, Disney has used this theater to show previews of upcoming movies.
Now, like many other attractions in The Studios, One Man's Dream is closed down. It will reopen in mid-September as Walt Disney Presents. This new attraction will continue to showcase Disney memorabilia, but it will also act as a preview center for what's to come in Disney World. It will be similar to the Walt Disney Imagineering Blue Sky Cellar in Disneyland's California Adventure, which has a rotating exhibit of new things coming to Disneyland.
So what can we expect to see when Walt Disney Presents reopens? Let's take a look.
One Man's Dream is now closed and will reopen as a preview center called Walt Disney Presents.
Toy Story Land is slated to open in the summer of 2018, but right now we know very little about the new area. That will change very soon when Walt Disney Presents has a complete model of the land on display. There will also be physical models of ride vehicles for the Slinky Dog Dash family coaster and the Alien Saucers plus concept art for the land.
We know the name for the new Star Wars Land; it will be called Galaxy's Edge. While a complete model was on display at the 2017 D23 Expo event, only a partial model will be seen at Walt Disney Presents.
This land, which is expected to open in 2019 after Disneyland's version, is said to be the most immersive land that Disney has ever created.
Of less consequence but equally noticeable, Mickey and Minnie's Runaway Railway will also be recognized in Walt Disney Presents. This attraction will replace The Great Movie Ride, which recently closed down. The new attraction was announced at D23 at the disappointment of many Great Movie Ride fans. However, I hope we will get a fitting replacement honoring Mickey.
Will you miss visiting One Man's Dream?